2024-03-29T10:23:56.594 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T10:23:56.594 DEBUG:teuthology.run:Teuthology command: teuthology --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --verbose --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622642 --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} --owner scheduled_amathuri@teuthology -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622642/orig.config.yaml 2024-03-29T10:23:56.626 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:23:56.704 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622642 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622642' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi084.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMXl6Z9hk+MhmxBMtW9InRHY/lu+i+im1gzZI1WaqF17XERyUg+tOpwSRjG/+63JuJhTlOL2azv4V46SqomePKE= smithi086.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE0cRLpkX/ZRNRgRxmItsxmFa4K2SATDuO+KjNERXWH7L6Wu1GBemAyH1YSSIGIkyoPht418Dx+hVVZEPMw6gOg= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T10:23:56.704 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T10:23:56.705 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T10:23:56.705 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T10:23:56.707 INFO:teuthology.task.internal:Checking packages... 2024-03-29T10:23:56.729 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T10:23:56.729 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T10:23:56.729 INFO:teuthology.packaging:ref: None 2024-03-29T10:23:56.729 INFO:teuthology.packaging:tag: None 2024-03-29T10:23:56.729 INFO:teuthology.packaging:branch: main 2024-03-29T10:23:56.730 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:23:56.730 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T10:23:56.997 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T10:23:56.998 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T10:23:57.005 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T10:23:57.005 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T10:23:57.064 INFO:teuthology.task.internal:Saving configuration 2024-03-29T10:23:57.077 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T10:23:57.083 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:23:57.103 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi084.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622642', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:18:50.477775', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMXl6Z9hk+MhmxBMtW9InRHY/lu+i+im1gzZI1WaqF17XERyUg+tOpwSRjG/+63JuJhTlOL2azv4V46SqomePKE='} 2024-03-29T10:23:57.120 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622642', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:18:50.476354', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE0cRLpkX/ZRNRgRxmItsxmFa4K2SATDuO+KjNERXWH7L6Wu1GBemAyH1YSSIGIkyoPht418Dx+hVVZEPMw6gOg='} 2024-03-29T10:23:57.120 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T10:23:57.127 INFO:teuthology.task.internal:roles: ubuntu@smithi084.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T10:23:57.127 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T10:23:57.127 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T10:23:57.202 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f720db3c0d0>, signals=[15]) 2024-03-29T10:23:57.202 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T10:23:57.208 INFO:teuthology.task.internal:Opening connections... 2024-03-29T10:23:57.208 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi084.front.sepia.ceph.com 2024-03-29T10:23:57.210 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:23:57.303 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi086.front.sepia.ceph.com 2024-03-29T10:23:57.304 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:23:57.392 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T10:23:57.399 DEBUG:teuthology.orchestra.run.smithi084:> uname -m 2024-03-29T10:23:57.416 INFO:teuthology.orchestra.run.smithi084.stdout:x86_64 2024-03-29T10:23:57.416 DEBUG:teuthology.orchestra.run.smithi084:> cat /etc/os-release 2024-03-29T10:23:57.473 INFO:teuthology.orchestra.run.smithi084.stdout:NAME="CentOS Stream" 2024-03-29T10:23:57.474 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION="9" 2024-03-29T10:23:57.474 INFO:teuthology.orchestra.run.smithi084.stdout:ID="centos" 2024-03-29T10:23:57.474 INFO:teuthology.orchestra.run.smithi084.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:23:57.474 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_ID="9" 2024-03-29T10:23:57.474 INFO:teuthology.orchestra.run.smithi084.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:23:57.474 INFO:teuthology.orchestra.run.smithi084.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:23:57.474 INFO:teuthology.orchestra.run.smithi084.stdout:ANSI_COLOR="0;31" 2024-03-29T10:23:57.474 INFO:teuthology.orchestra.run.smithi084.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:23:57.474 INFO:teuthology.orchestra.run.smithi084.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:23:57.474 INFO:teuthology.orchestra.run.smithi084.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:23:57.474 INFO:teuthology.orchestra.run.smithi084.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:23:57.474 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:23:57.474 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:23:57.475 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2024-03-29T10:23:57.498 DEBUG:teuthology.orchestra.run.smithi086:> uname -m 2024-03-29T10:23:57.516 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2024-03-29T10:23:57.516 DEBUG:teuthology.orchestra.run.smithi086:> cat /etc/os-release 2024-03-29T10:23:57.573 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="CentOS Stream" 2024-03-29T10:23:57.573 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="9" 2024-03-29T10:23:57.573 INFO:teuthology.orchestra.run.smithi086.stdout:ID="centos" 2024-03-29T10:23:57.573 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:23:57.573 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="9" 2024-03-29T10:23:57.573 INFO:teuthology.orchestra.run.smithi086.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:23:57.573 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:23:57.574 INFO:teuthology.orchestra.run.smithi086.stdout:ANSI_COLOR="0;31" 2024-03-29T10:23:57.574 INFO:teuthology.orchestra.run.smithi086.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:23:57.574 INFO:teuthology.orchestra.run.smithi086.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:23:57.574 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:23:57.574 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:23:57.574 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:23:57.574 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:23:57.574 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2024-03-29T10:23:57.598 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T10:23:57.609 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T10:23:57.621 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T10:23:57.621 DEBUG:teuthology.orchestra.run.smithi084:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:23:57.623 DEBUG:teuthology.orchestra.run.smithi086:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:23:57.639 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T10:23:57.649 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T10:23:57.650 DEBUG:teuthology.orchestra.run.smithi084:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:23:57.680 DEBUG:teuthology.orchestra.run.smithi086:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:23:57.791 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T10:23:57.848 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T10:23:57.866 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T10:23:57.866 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T10:23:57.866 DEBUG:teuthology.orchestra.run.smithi084:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:23:57.867 DEBUG:teuthology.orchestra.run.smithi086:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:23:57.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:23:57.883 DEBUG:teuthology.orchestra.run.smithi084:> uname -r 2024-03-29T10:23:57.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:23:57.884 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-03-29T10:23:57.939 INFO:teuthology.orchestra.run.smithi084.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:23:57.940 INFO:teuthology.task.kernel:Running kernel on smithi084: 5.14.0-361.el9.x86_64 2024-03-29T10:23:57.940 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum install -y kernel 2024-03-29T10:23:57.941 INFO:teuthology.orchestra.run.smithi086.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:23:57.942 INFO:teuthology.task.kernel:Running kernel on smithi086: 5.14.0-361.el9.x86_64 2024-03-29T10:23:57.942 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum install -y kernel 2024-03-29T10:23:59.942 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-29T10:24:00.036 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-29T10:24:01.478 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2024-03-29T10:24:01.592 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-03-29T10:24:02.724 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-03-29T10:24:05.585 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - BaseOS 2.1 MB/s | 8.0 MB 00:03 2024-03-29T10:24:07.175 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - AppStream 5.6 MB/s | 19 MB 00:03 2024-03-29T10:24:11.850 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - CRB 5.0 MB/s | 6.1 MB 00:01 2024-03-29T10:24:14.400 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 16 kB 00:01 2024-03-29T10:24:15.768 INFO:teuthology.orchestra.run.smithi084.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-29T10:24:20.701 INFO:teuthology.orchestra.run.smithi084.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-29T10:24:22.966 INFO:teuthology.orchestra.run.smithi084.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:24:22.966 INFO:teuthology.orchestra.run.smithi084.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:24:23.071 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T10:24:23.082 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:24:23.082 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-29T10:24:23.082 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:24:23.082 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T10:24:23.082 INFO:teuthology.orchestra.run.smithi084.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:24:23.082 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-29T10:24:23.083 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:24:23.083 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:24:23.083 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:24:23.083 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:24:23.083 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T10:24:23.083 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:24:23.083 INFO:teuthology.orchestra.run.smithi084.stdout:Install 4 Packages 2024-03-29T10:24:23.083 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:24:23.084 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 76 M 2024-03-29T10:24:23.084 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 123 M 2024-03-29T10:24:23.084 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T10:24:23.260 INFO:teuthology.orchestra.run.smithi084.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-29T10:24:24.711 INFO:teuthology.orchestra.run.smithi084.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 9.3 MB/s | 15 MB 00:01 2024-03-29T10:24:25.644 INFO:teuthology.orchestra.run.smithi084.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 13 MB/s | 34 MB 00:02 2024-03-29T10:24:26.069 INFO:teuthology.orchestra.run.smithi084.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 9.8 MB/s | 28 MB 00:02 2024-03-29T10:24:26.072 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:24:26.073 INFO:teuthology.orchestra.run.smithi084.stdout:Total 26 MB/s | 76 MB 00:02 2024-03-29T10:24:26.073 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T10:24:26.452 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T10:24:26.453 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T10:24:27.283 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T10:24:27.284 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T10:24:28.525 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - AppStream 885 kB/s | 19 MB 00:21 2024-03-29T10:24:29.597 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T10:24:30.193 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:24:30.286 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:24:33.026 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - CRB 6.7 MB/s | 6.1 MB 00:00 2024-03-29T10:24:33.033 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:24:33.164 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:24:34.685 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 16 kB 00:00 2024-03-29T10:24:36.014 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-29T10:24:37.855 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:24:38.090 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:24:41.186 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-29T10:24:42.786 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:24:43.451 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:24:43.451 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:24:43.550 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T10:24:43.556 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:24:43.556 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-29T10:24:43.556 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:24:43.556 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T10:24:43.556 INFO:teuthology.orchestra.run.smithi086.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:24:43.556 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-29T10:24:43.557 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:24:43.557 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:24:43.557 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:24:43.557 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:24:43.557 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T10:24:43.557 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:24:43.557 INFO:teuthology.orchestra.run.smithi086.stdout:Install 4 Packages 2024-03-29T10:24:43.557 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:24:43.558 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 76 M 2024-03-29T10:24:43.558 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 123 M 2024-03-29T10:24:43.558 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T10:24:43.729 INFO:teuthology.orchestra.run.smithi086.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-29T10:24:44.330 INFO:teuthology.orchestra.run.smithi086.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 20 MB/s | 15 MB 00:00 2024-03-29T10:24:44.955 INFO:teuthology.orchestra.run.smithi086.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 24 MB/s | 34 MB 00:01 2024-03-29T10:24:45.247 INFO:teuthology.orchestra.run.smithi086.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 18 MB/s | 28 MB 00:01 2024-03-29T10:24:45.249 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:24:45.250 INFO:teuthology.orchestra.run.smithi086.stdout:Total 45 MB/s | 76 MB 00:01 2024-03-29T10:24:45.250 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T10:24:45.616 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T10:24:45.616 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T10:24:46.468 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T10:24:46.469 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T10:24:48.779 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T10:24:49.364 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:24:49.447 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:24:52.176 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:24:52.306 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:24:57.021 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:24:57.080 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:25:01.823 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:25:12.167 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:25:12.168 INFO:teuthology.orchestra.run.smithi084.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:25:12.168 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:25:12.211 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:25:12.552 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:25:12.552 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:25:12.552 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:25:12.552 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:25:12.986 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:25:12.986 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:25:12.986 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T10:25:12.986 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:25:12.986 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:25:12.986 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:25:12.986 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:25:12.986 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:25:12.986 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T10:25:13.351 DEBUG:teuthology.orchestra.run.smithi084:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:25:13.449 INFO:teuthology.orchestra.run.smithi084.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:25:13.449 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:25:13.450 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:25:13.450 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:25:13.450 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:25:13.450 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T10:25:13.450 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi084.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:25:13.450 DEBUG:teuthology.orchestra.run.smithi084:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:25:13.530 INFO:teuthology.orchestra.run.smithi084.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:25:13.530 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:25:13.531 DEBUG:teuthology.orchestra.run.smithi084:> sudo rpm -qi grub2-tools 2024-03-29T10:25:13.573 INFO:teuthology.orchestra.run.smithi084.stdout:Name : grub2-tools 2024-03-29T10:25:13.573 INFO:teuthology.orchestra.run.smithi084.stdout:Epoch : 1 2024-03-29T10:25:13.573 INFO:teuthology.orchestra.run.smithi084.stdout:Version : 2.06 2024-03-29T10:25:13.573 INFO:teuthology.orchestra.run.smithi084.stdout:Release : 68.el9 2024-03-29T10:25:13.573 INFO:teuthology.orchestra.run.smithi084.stdout:Architecture: x86_64 2024-03-29T10:25:13.573 INFO:teuthology.orchestra.run.smithi084.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:25:13.573 INFO:teuthology.orchestra.run.smithi084.stdout:Group : Unspecified 2024-03-29T10:25:13.573 INFO:teuthology.orchestra.run.smithi084.stdout:Size : 8264254 2024-03-29T10:25:13.573 INFO:teuthology.orchestra.run.smithi084.stdout:License : GPLv3+ 2024-03-29T10:25:13.573 INFO:teuthology.orchestra.run.smithi084.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:25:13.574 INFO:teuthology.orchestra.run.smithi084.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:25:13.574 INFO:teuthology.orchestra.run.smithi084.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:25:13.574 INFO:teuthology.orchestra.run.smithi084.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:25:13.574 INFO:teuthology.orchestra.run.smithi084.stdout:Packager : builder@centos.org 2024-03-29T10:25:13.574 INFO:teuthology.orchestra.run.smithi084.stdout:Vendor : CentOS 2024-03-29T10:25:13.574 INFO:teuthology.orchestra.run.smithi084.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:25:13.574 INFO:teuthology.orchestra.run.smithi084.stdout:Summary : Support tools for GRUB. 2024-03-29T10:25:13.574 INFO:teuthology.orchestra.run.smithi084.stdout:Description : 2024-03-29T10:25:13.574 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:25:13.574 INFO:teuthology.orchestra.run.smithi084.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:25:13.574 INFO:teuthology.orchestra.run.smithi084.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:25:13.574 INFO:teuthology.orchestra.run.smithi084.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:25:13.574 INFO:teuthology.orchestra.run.smithi084.stdout:hardware devices. 2024-03-29T10:25:13.575 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:25:13.575 INFO:teuthology.orchestra.run.smithi084.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:25:13.575 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:25:13.575 INFO:teuthology.task.kernel:Updating grub on smithi084 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:25:13.576 DEBUG:teuthology.orchestra.run.smithi084:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:25:14.226 INFO:teuthology.orchestra.run.smithi084.stderr:Generating grub configuration file ... 2024-03-29T10:25:15.632 INFO:teuthology.orchestra.run.smithi084.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:25:15.669 INFO:teuthology.orchestra.run.smithi084.stderr:done 2024-03-29T10:25:15.676 DEBUG:teuthology.orchestra.run.smithi084:> mktemp 2024-03-29T10:25:15.691 INFO:teuthology.orchestra.run.smithi084.stdout:/tmp/tmp.cp3852WXMv 2024-03-29T10:25:15.692 DEBUG:teuthology.orchestra.run.smithi084:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.cp3852WXMv 2024-03-29T10:25:15.759 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0666 /tmp/tmp.cp3852WXMv 2024-03-29T10:25:15.894 DEBUG:teuthology.orchestra.remote:smithi084:/tmp/tmp.cp3852WXMv is 6KB 2024-03-29T10:25:15.943 DEBUG:teuthology.orchestra.run.smithi084:> rm -fr /tmp/tmp.cp3852WXMv 2024-03-29T10:25:15.957 DEBUG:teuthology.orchestra.run.smithi084:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:25:16.030 INFO:teuthology.orchestra.run.smithi084.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:25:16.030 INFO:teuthology.orchestra.run.smithi084.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:25:16.030 INFO:teuthology.orchestra.run.smithi084.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:25:16.030 INFO:teuthology.orchestra.run.smithi084.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:25:16.031 DEBUG:teuthology.orchestra.run.smithi084:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:25:16.183 DEBUG:teuthology.orchestra.run.smithi084:> sudo shutdown -r now 2024-03-29T10:25:31.180 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:25:31.180 INFO:teuthology.orchestra.run.smithi086.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:25:31.180 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:25:31.224 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:25:31.597 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:25:31.597 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:25:31.597 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:25:31.597 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:25:31.999 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:25:31.999 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:25:31.999 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T10:25:31.999 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:25:32.000 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:25:32.000 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:25:32.000 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:25:32.000 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:25:32.000 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T10:25:32.322 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:25:32.437 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:25:32.438 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:25:32.438 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:25:32.438 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:25:32.438 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:25:32.438 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T10:25:32.438 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi086.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:25:32.438 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:25:32.518 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:25:32.519 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:25:32.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo rpm -qi grub2-tools 2024-03-29T10:25:32.553 INFO:teuthology.orchestra.run.smithi086.stdout:Name : grub2-tools 2024-03-29T10:25:32.553 INFO:teuthology.orchestra.run.smithi086.stdout:Epoch : 1 2024-03-29T10:25:32.553 INFO:teuthology.orchestra.run.smithi086.stdout:Version : 2.06 2024-03-29T10:25:32.553 INFO:teuthology.orchestra.run.smithi086.stdout:Release : 68.el9 2024-03-29T10:25:32.553 INFO:teuthology.orchestra.run.smithi086.stdout:Architecture: x86_64 2024-03-29T10:25:32.553 INFO:teuthology.orchestra.run.smithi086.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:25:32.553 INFO:teuthology.orchestra.run.smithi086.stdout:Group : Unspecified 2024-03-29T10:25:32.554 INFO:teuthology.orchestra.run.smithi086.stdout:Size : 8264254 2024-03-29T10:25:32.554 INFO:teuthology.orchestra.run.smithi086.stdout:License : GPLv3+ 2024-03-29T10:25:32.554 INFO:teuthology.orchestra.run.smithi086.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:25:32.554 INFO:teuthology.orchestra.run.smithi086.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:25:32.554 INFO:teuthology.orchestra.run.smithi086.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:25:32.554 INFO:teuthology.orchestra.run.smithi086.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:25:32.554 INFO:teuthology.orchestra.run.smithi086.stdout:Packager : builder@centos.org 2024-03-29T10:25:32.554 INFO:teuthology.orchestra.run.smithi086.stdout:Vendor : CentOS 2024-03-29T10:25:32.554 INFO:teuthology.orchestra.run.smithi086.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:25:32.554 INFO:teuthology.orchestra.run.smithi086.stdout:Summary : Support tools for GRUB. 2024-03-29T10:25:32.554 INFO:teuthology.orchestra.run.smithi086.stdout:Description : 2024-03-29T10:25:32.554 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:25:32.554 INFO:teuthology.orchestra.run.smithi086.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:25:32.554 INFO:teuthology.orchestra.run.smithi086.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:25:32.555 INFO:teuthology.orchestra.run.smithi086.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:25:32.555 INFO:teuthology.orchestra.run.smithi086.stdout:hardware devices. 2024-03-29T10:25:32.555 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:25:32.555 INFO:teuthology.orchestra.run.smithi086.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:25:32.556 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:25:32.556 INFO:teuthology.task.kernel:Updating grub on smithi086 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:25:32.556 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:25:33.205 INFO:teuthology.orchestra.run.smithi086.stderr:Generating grub configuration file ... 2024-03-29T10:25:34.633 INFO:teuthology.orchestra.run.smithi086.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:25:34.671 INFO:teuthology.orchestra.run.smithi086.stderr:done 2024-03-29T10:25:34.675 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2024-03-29T10:25:34.736 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.RCqtkE0aQx 2024-03-29T10:25:34.736 DEBUG:teuthology.orchestra.run.smithi086:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.RCqtkE0aQx 2024-03-29T10:25:34.804 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.RCqtkE0aQx 2024-03-29T10:25:34.939 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.RCqtkE0aQx is 6KB 2024-03-29T10:25:34.958 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.RCqtkE0aQx 2024-03-29T10:25:34.972 DEBUG:teuthology.orchestra.run.smithi086:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:25:35.039 INFO:teuthology.orchestra.run.smithi086.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:25:35.039 INFO:teuthology.orchestra.run.smithi086.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:25:35.039 INFO:teuthology.orchestra.run.smithi086.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:25:35.039 INFO:teuthology.orchestra.run.smithi086.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:25:35.041 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:25:35.197 DEBUG:teuthology.orchestra.run.smithi086:> sudo shutdown -r now 2024-03-29T10:25:46.194 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T10:25:46.195 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-03-29T10:25:46.195 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:26:04.702 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2024-03-29T10:26:05.200 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T10:26:05.201 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-03-29T10:26:05.201 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:26:13.711 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-03-29T10:26:13.713 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:26:16.770 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2024-03-29T10:26:28.782 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-03-29T10:26:28.783 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:26:47.202 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.84 2024-03-29T10:27:02.218 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-03-29T10:27:02.219 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:27:02.606 DEBUG:teuthology.orchestra.run.smithi084:> true 2024-03-29T10:27:03.019 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-03-29T10:27:03.019 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:27:03.019 DEBUG:teuthology.orchestra.run.smithi084:> uname -r 2024-03-29T10:27:03.033 INFO:teuthology.orchestra.run.smithi084.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:27:03.033 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:27:03.033 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:27:03.033 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:27:04.034 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T10:27:04.034 DEBUG:teuthology.orchestra.run.smithi084:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:27:04.096 INFO:teuthology.orchestra.run.smithi084.stdout:ttyS1 2024-03-29T10:27:04.124 DEBUG:teuthology.parallel:result is None 2024-03-29T10:27:05.206 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T10:27:14.210 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-03-29T10:27:14.212 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:27:17.749 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-03-29T10:27:18.276 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-03-29T10:27:18.276 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:27:18.276 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-03-29T10:27:18.291 INFO:teuthology.orchestra.run.smithi086.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:27:18.291 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:27:18.291 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:27:18.291 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:27:19.292 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T10:27:19.292 DEBUG:teuthology.orchestra.run.smithi086:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:27:19.346 INFO:teuthology.orchestra.run.smithi086.stdout:ttyS1 2024-03-29T10:27:19.389 DEBUG:teuthology.parallel:result is None 2024-03-29T10:27:19.390 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T10:27:19.399 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T10:27:19.399 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:27:19.402 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:27:19.418 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T10:27:19.425 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T10:27:19.439 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T10:27:19.439 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:27:19.461 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:27:19.603 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T10:27:19.615 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T10:27:19.615 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:27:19.617 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:27:19.648 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:27:19.649 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:27:19.662 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:27:19.662 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:27:19.664 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T10:27:19.671 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T10:27:19.672 DEBUG:teuthology.orchestra.run.smithi084:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:27:19.706 DEBUG:teuthology.orchestra.run.smithi086:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:27:19.734 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T10:27:19.745 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T10:27:19.746 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:27:19.777 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:27:19.791 DEBUG:teuthology.orchestra.run.smithi084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:27:19.873 DEBUG:teuthology.orchestra.run.smithi084:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:27:19.939 DEBUG:teuthology.orchestra.run.smithi084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:27:19.981 DEBUG:teuthology.orchestra.run.smithi084:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:27:20.024 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T10:27:20.025 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:27:20.092 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:27:20.134 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:27:20.187 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:27:20.272 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:27:20.312 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T10:27:20.312 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:27:20.383 DEBUG:teuthology.orchestra.run.smithi084:> sudo service rsyslog restart 2024-03-29T10:27:20.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo service rsyslog restart 2024-03-29T10:27:20.431 INFO:teuthology.orchestra.run.smithi084.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:27:20.476 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:27:20.804 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T10:27:20.815 INFO:teuthology.task.internal:Starting timer... 2024-03-29T10:27:20.815 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T10:27:20.832 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T10:27:20.847 DEBUG:teuthology.orchestra.run.smithi084:> sudo service auditd rotate 2024-03-29T10:27:20.908 INFO:teuthology.orchestra.run.smithi084.stdout:Rotating logs: 2024-03-29T10:27:20.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo service auditd rotate 2024-03-29T10:27:20.971 INFO:teuthology.orchestra.run.smithi086.stdout:Rotating logs: 2024-03-29T10:27:20.973 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T10:27:20.973 DEBUG:teuthology.orchestra.run.smithi084:> /usr/sbin/getenforce 2024-03-29T10:27:21.000 INFO:teuthology.orchestra.run.smithi084.stdout:Permissive 2024-03-29T10:27:21.000 DEBUG:teuthology.orchestra.run.smithi086:> /usr/sbin/getenforce 2024-03-29T10:27:21.028 INFO:teuthology.orchestra.run.smithi086.stdout:Permissive 2024-03-29T10:27:21.028 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi084.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi086.front.sepia.ceph.com': 'permissive'} 2024-03-29T10:27:21.028 DEBUG:teuthology.orchestra.run.smithi084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:27:21.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:27:21.069 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:27:21.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:27:21.096 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T10:27:21.096 DEBUG:teuthology.orchestra.run.smithi084:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:27:21.152 DEBUG:teuthology.orchestra.run.smithi086:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:27:21.188 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T10:27:21.199 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T10:27:21.216 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T10:27:21.218 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi084.front.sepia.ceph.com,smithi086.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T10:31:45.955 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi084.front.sepia.ceph.com'), Remote(name='ubuntu@smithi086.front.sepia.ceph.com')] 2024-03-29T10:31:45.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-03-29T10:31:45.959 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:31:46.050 DEBUG:teuthology.orchestra.run.smithi084:> true 2024-03-29T10:31:46.125 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-03-29T10:31:46.126 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-03-29T10:31:46.126 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:31:46.213 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-03-29T10:31:46.292 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-03-29T10:31:46.292 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T10:31:46.303 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T10:31:46.303 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:31:46.304 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:31:46.306 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:31:46.306 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:31:46.335 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:31:46.352 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:31:46.363 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:31:46.379 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:31:46.381 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: ntpd: command not found 2024-03-29T10:31:46.394 INFO:teuthology.orchestra.run.smithi084.stdout:506 Cannot talk to daemon 2024-03-29T10:31:46.409 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: ntpd: command not found 2024-03-29T10:31:46.410 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:31:46.422 INFO:teuthology.orchestra.run.smithi086.stdout:506 Cannot talk to daemon 2024-03-29T10:31:46.427 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:31:46.439 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:31:46.455 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:31:46.483 INFO:teuthology.orchestra.run.smithi084.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:31:46.486 INFO:teuthology.orchestra.run.smithi084.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:31:46.486 INFO:teuthology.orchestra.run.smithi084.stdout:=============================================================================== 2024-03-29T10:31:46.486 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:31:46.486 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:31:46.486 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:31:46.486 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:31:46.512 INFO:teuthology.orchestra.run.smithi086.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:31:46.516 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:31:46.516 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-03-29T10:31:46.516 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:31:46.516 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:31:46.516 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:31:46.516 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:31:46.517 INFO:teuthology.run_tasks:Running task install... 2024-03-29T10:31:46.529 DEBUG:teuthology.task.install:project ceph 2024-03-29T10:31:46.529 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T10:31:46.530 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T10:31:46.530 INFO:teuthology.task.install:Using flavor: default 2024-03-29T10:31:46.540 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T10:31:46.540 INFO:teuthology.task.install:extra packages: [] 2024-03-29T10:31:46.540 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:31:46.541 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:31:46.543 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:31:46.543 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:31:46.693 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:31:46.693 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:31:46.701 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:31:46.701 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:31:46.992 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:31:46.993 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T10:31:46.993 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:31:47.020 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:31:47.021 DEBUG:teuthology.orchestra.run.smithi084:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:31:47.054 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:31:47.055 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T10:31:47.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:31:47.083 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:31:47.083 DEBUG:teuthology.orchestra.run.smithi086:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:31:47.088 DEBUG:teuthology.orchestra.run.smithi084:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:31:47.151 DEBUG:teuthology.orchestra.run.smithi086:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:31:47.168 DEBUG:teuthology.orchestra.run.smithi084:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:31:47.231 DEBUG:teuthology.orchestra.run.smithi086:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:31:47.235 INFO:teuthology.orchestra.run.smithi084.stdout:check_obsoletes = 1 2024-03-29T10:31:47.237 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum clean all 2024-03-29T10:31:47.299 INFO:teuthology.orchestra.run.smithi086.stdout:check_obsoletes = 1 2024-03-29T10:31:47.300 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum clean all 2024-03-29T10:31:47.484 INFO:teuthology.orchestra.run.smithi084.stdout:74 files removed 2024-03-29T10:31:47.510 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-radosgw 2024-03-29T10:31:47.547 INFO:teuthology.orchestra.run.smithi086.stdout:74 files removed 2024-03-29T10:31:47.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-radosgw 2024-03-29T10:31:48.424 INFO:teuthology.orchestra.run.smithi084.stdout:ceph packages for x86_64 133 kB/s | 82 kB 00:00 2024-03-29T10:31:48.489 INFO:teuthology.orchestra.run.smithi086.stdout:ceph packages for x86_64 133 kB/s | 82 kB 00:00 2024-03-29T10:31:49.083 INFO:teuthology.orchestra.run.smithi084.stdout:ceph noarch packages 16 kB/s | 10 kB 00:00 2024-03-29T10:31:49.131 INFO:teuthology.orchestra.run.smithi086.stdout:ceph noarch packages 17 kB/s | 10 kB 00:00 2024-03-29T10:31:49.759 INFO:teuthology.orchestra.run.smithi084.stdout:ceph source packages 3.0 kB/s | 1.9 kB 00:00 2024-03-29T10:31:49.815 INFO:teuthology.orchestra.run.smithi086.stdout:ceph source packages 2.9 kB/s | 1.9 kB 00:00 2024-03-29T10:31:50.169 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-29T10:31:50.284 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-29T10:31:51.620 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS-9 - BaseOS 5.9 MB/s | 2.0 MB 00:00 2024-03-29T10:31:51.720 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.0 MB 00:00 2024-03-29T10:31:53.752 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.0 MB 00:01 2024-03-29T10:31:55.477 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - BaseOS 2.3 MB/s | 8.0 MB 00:03 2024-03-29T10:31:55.820 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-03-29T10:31:59.044 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - AppStream 7.5 MB/s | 19 MB 00:02 2024-03-29T10:32:03.161 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - CRB 9.3 MB/s | 6.1 MB 00:00 2024-03-29T10:32:03.770 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - CRB 1.4 MB/s | 6.1 MB 00:04 2024-03-29T10:32:04.670 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 16 kB 00:00 2024-03-29T10:32:05.963 INFO:teuthology.orchestra.run.smithi084.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-03-29T10:32:06.263 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-03-29T10:32:07.664 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-29T10:32:10.821 INFO:teuthology.orchestra.run.smithi084.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-29T10:32:12.722 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-29T10:32:13.380 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T10:32:13.391 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:32:13.391 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-29T10:32:13.391 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:32:13.391 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T10:32:13.391 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:32:13.391 INFO:teuthology.orchestra.run.smithi084.stdout:Upgrading: 2024-03-29T10:32:13.391 INFO:teuthology.orchestra.run.smithi084.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:32:13.391 INFO:teuthology.orchestra.run.smithi084.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:32:13.391 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-29T10:32:13.391 INFO:teuthology.orchestra.run.smithi084.stdout: boost-program-options 2024-03-29T10:32:13.392 INFO:teuthology.orchestra.run.smithi084.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:32:13.392 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:32:13.392 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:32:13.392 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:32:13.392 INFO:teuthology.orchestra.run.smithi084.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:32:13.392 INFO:teuthology.orchestra.run.smithi084.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:32:13.392 INFO:teuthology.orchestra.run.smithi084.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:32:13.392 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:32:13.392 INFO:teuthology.orchestra.run.smithi084.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:32:13.392 INFO:teuthology.orchestra.run.smithi084.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:32:13.392 INFO:teuthology.orchestra.run.smithi084.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:32:13.392 INFO:teuthology.orchestra.run.smithi084.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:32:13.392 INFO:teuthology.orchestra.run.smithi084.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:32:13.392 INFO:teuthology.orchestra.run.smithi084.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:32:13.393 INFO:teuthology.orchestra.run.smithi084.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:32:13.393 INFO:teuthology.orchestra.run.smithi084.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:32:13.393 INFO:teuthology.orchestra.run.smithi084.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:32:13.393 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-argparse 2024-03-29T10:32:13.393 INFO:teuthology.orchestra.run.smithi084.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:32:13.393 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:32:13.393 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:32:13.393 INFO:teuthology.orchestra.run.smithi084.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:32:13.393 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:32:13.393 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:32:13.393 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:32:13.393 INFO:teuthology.orchestra.run.smithi084.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:32:13.393 INFO:teuthology.orchestra.run.smithi084.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:32:13.393 INFO:teuthology.orchestra.run.smithi084.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:32:13.394 INFO:teuthology.orchestra.run.smithi084.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:32:13.394 INFO:teuthology.orchestra.run.smithi084.stdout:Installing weak dependencies: 2024-03-29T10:32:13.394 INFO:teuthology.orchestra.run.smithi084.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:32:13.394 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:32:13.394 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T10:32:13.394 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:32:13.394 INFO:teuthology.orchestra.run.smithi084.stdout:Install 30 Packages 2024-03-29T10:32:13.394 INFO:teuthology.orchestra.run.smithi084.stdout:Upgrade 2 Packages 2024-03-29T10:32:13.394 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:32:13.395 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 58 M 2024-03-29T10:32:13.395 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T10:32:15.320 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T10:32:15.328 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:32:15.328 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-29T10:32:15.328 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:32:15.328 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T10:32:15.329 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:32:15.329 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrading: 2024-03-29T10:32:15.329 INFO:teuthology.orchestra.run.smithi086.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:32:15.329 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:32:15.329 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-29T10:32:15.329 INFO:teuthology.orchestra.run.smithi086.stdout: boost-program-options 2024-03-29T10:32:15.329 INFO:teuthology.orchestra.run.smithi086.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:32:15.329 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:32:15.329 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:32:15.329 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:32:15.329 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:32:15.329 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:32:15.329 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:32:15.329 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:32:15.330 INFO:teuthology.orchestra.run.smithi086.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:32:15.330 INFO:teuthology.orchestra.run.smithi086.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:32:15.330 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:32:15.330 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:32:15.330 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:32:15.330 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:32:15.330 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:32:15.330 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:32:15.330 INFO:teuthology.orchestra.run.smithi086.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:32:15.330 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse 2024-03-29T10:32:15.330 INFO:teuthology.orchestra.run.smithi086.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:32:15.330 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:32:15.330 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:32:15.331 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:32:15.331 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:32:15.331 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:32:15.331 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:32:15.331 INFO:teuthology.orchestra.run.smithi086.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:32:15.331 INFO:teuthology.orchestra.run.smithi086.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:32:15.331 INFO:teuthology.orchestra.run.smithi086.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:32:15.331 INFO:teuthology.orchestra.run.smithi086.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:32:15.331 INFO:teuthology.orchestra.run.smithi086.stdout:Installing weak dependencies: 2024-03-29T10:32:15.331 INFO:teuthology.orchestra.run.smithi086.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:32:15.331 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:32:15.331 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T10:32:15.331 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:32:15.331 INFO:teuthology.orchestra.run.smithi086.stdout:Install 30 Packages 2024-03-29T10:32:15.332 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrade 2 Packages 2024-03-29T10:32:15.332 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:32:15.332 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 58 M 2024-03-29T10:32:15.332 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T10:32:15.338 INFO:teuthology.orchestra.run.smithi084.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 2.8 MB/s | 5.3 MB 00:01 2024-03-29T10:32:15.388 INFO:teuthology.orchestra.run.smithi084.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 505 kB/s | 25 kB 00:00 2024-03-29T10:32:15.489 INFO:teuthology.orchestra.run.smithi084.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 7.2 MB/s | 742 kB 00:00 2024-03-29T10:32:15.564 INFO:teuthology.orchestra.run.smithi084.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 6.5 MB/s | 498 kB 00:00 2024-03-29T10:32:16.123 INFO:teuthology.orchestra.run.smithi084.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 9.1 MB/s | 5.1 MB 00:00 2024-03-29T10:32:16.189 INFO:teuthology.orchestra.run.smithi084.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 680 kB/s | 45 kB 00:00 2024-03-29T10:32:16.256 INFO:teuthology.orchestra.run.smithi084.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.9 MB/s | 131 kB 00:00 2024-03-29T10:32:16.332 INFO:teuthology.orchestra.run.smithi084.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 2.1 MB/s | 162 kB 00:00 2024-03-29T10:32:16.415 INFO:teuthology.orchestra.run.smithi084.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 3.8 MB/s | 320 kB 00:00 2024-03-29T10:32:16.499 INFO:teuthology.orchestra.run.smithi084.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 3.5 MB/s | 298 kB 00:00 2024-03-29T10:32:16.541 INFO:teuthology.orchestra.run.smithi084.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 2.3 MB/s | 99 kB 00:00 2024-03-29T10:32:16.649 INFO:teuthology.orchestra.run.smithi084.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 978 kB/s | 106 kB 00:00 2024-03-29T10:32:16.700 INFO:teuthology.orchestra.run.smithi084.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-29T10:32:16.725 INFO:teuthology.orchestra.run.smithi084.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-29T10:32:16.731 INFO:teuthology.orchestra.run.smithi086.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 4.0 MB/s | 5.3 MB 00:01 2024-03-29T10:32:16.773 INFO:teuthology.orchestra.run.smithi086.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 605 kB/s | 25 kB 00:00 2024-03-29T10:32:16.783 INFO:teuthology.orchestra.run.smithi084.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-29T10:32:16.825 INFO:teuthology.orchestra.run.smithi084.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-29T10:32:16.859 INFO:teuthology.orchestra.run.smithi084.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-29T10:32:16.901 INFO:teuthology.orchestra.run.smithi084.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.1 MB/s | 220 kB 00:00 2024-03-29T10:32:16.926 INFO:teuthology.orchestra.run.smithi084.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.4 MB/s | 35 kB 00:00 2024-03-29T10:32:16.951 INFO:teuthology.orchestra.run.smithi084.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.8 MB/s | 46 kB 00:00 2024-03-29T10:32:16.974 INFO:teuthology.orchestra.run.smithi086.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 3.6 MB/s | 742 kB 00:00 2024-03-29T10:32:17.166 INFO:teuthology.orchestra.run.smithi086.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 2.5 MB/s | 498 kB 00:00 2024-03-29T10:32:17.393 INFO:teuthology.orchestra.run.smithi084.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 700 kB/s | 309 kB 00:00 2024-03-29T10:32:17.566 INFO:teuthology.orchestra.run.smithi086.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 4.5 MB/s | 9.7 MB 00:02 2024-03-29T10:32:17.658 INFO:teuthology.orchestra.run.smithi086.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 493 kB/s | 45 kB 00:00 2024-03-29T10:32:17.727 INFO:teuthology.orchestra.run.smithi084.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-03-29T10:32:17.794 INFO:teuthology.orchestra.run.smithi084.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 392 kB/s | 26 kB 00:00 2024-03-29T10:32:17.842 INFO:teuthology.orchestra.run.smithi086.stdout:(7/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 8.1 MB/s | 20 MB 00:02 2024-03-29T10:32:17.861 INFO:teuthology.orchestra.run.smithi084.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 731 kB/s | 49 kB 00:00 2024-03-29T10:32:17.867 INFO:teuthology.orchestra.run.smithi086.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 630 kB/s | 131 kB 00:00 2024-03-29T10:32:17.892 INFO:teuthology.orchestra.run.smithi086.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 3.2 MB/s | 162 kB 00:00 2024-03-29T10:32:17.921 INFO:teuthology.orchestra.run.smithi086.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 5.8 MB/s | 320 kB 00:00 2024-03-29T10:32:17.959 INFO:teuthology.orchestra.run.smithi086.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 4.4 MB/s | 298 kB 00:00 2024-03-29T10:32:17.961 INFO:teuthology.orchestra.run.smithi084.stdout:(25/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 2.2 MB/s | 9.7 MB 00:04 2024-03-29T10:32:17.984 INFO:teuthology.orchestra.run.smithi086.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.5 MB/s | 99 kB 00:00 2024-03-29T10:32:17.986 INFO:teuthology.orchestra.run.smithi084.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 538 kB/s | 67 kB 00:00 2024-03-29T10:32:18.060 INFO:teuthology.orchestra.run.smithi086.stdout:(13/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 5.7 MB/s | 5.1 MB 00:00 2024-03-29T10:32:18.061 INFO:teuthology.orchestra.run.smithi084.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-29T10:32:18.162 INFO:teuthology.orchestra.run.smithi084.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-03-29T10:32:18.185 INFO:teuthology.orchestra.run.smithi086.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 469 kB/s | 106 kB 00:00 2024-03-29T10:32:18.210 INFO:teuthology.orchestra.run.smithi086.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 705 kB/s | 159 kB 00:00 2024-03-29T10:32:18.235 INFO:teuthology.orchestra.run.smithi086.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 344 kB/s | 44 kB 00:00 2024-03-29T10:32:18.352 INFO:teuthology.orchestra.run.smithi086.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.9 MB/s | 661 kB 00:00 2024-03-29T10:32:18.378 INFO:teuthology.orchestra.run.smithi086.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 291 kB/s | 41 kB 00:00 2024-03-29T10:32:18.403 INFO:teuthology.orchestra.run.smithi086.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.7 MB/s | 282 kB 00:00 2024-03-29T10:32:18.428 INFO:teuthology.orchestra.run.smithi086.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.8 MB/s | 220 kB 00:00 2024-03-29T10:32:18.453 INFO:teuthology.orchestra.run.smithi086.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 465 kB/s | 35 kB 00:00 2024-03-29T10:32:18.479 INFO:teuthology.orchestra.run.smithi086.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 615 kB/s | 46 kB 00:00 2024-03-29T10:32:18.479 INFO:teuthology.orchestra.run.smithi084.stdout:(29/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 11 MB/s | 3.4 MB 00:00 2024-03-29T10:32:18.521 INFO:teuthology.orchestra.run.smithi084.stdout:(30/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.5 MB/s | 838 kB 00:00 2024-03-29T10:32:18.688 INFO:teuthology.orchestra.run.smithi084.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 15 MB/s | 3.1 MB 00:00 2024-03-29T10:32:18.771 INFO:teuthology.orchestra.run.smithi086.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 89 kB/s | 26 kB 00:00 2024-03-29T10:32:18.871 INFO:teuthology.orchestra.run.smithi086.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 741 kB/s | 309 kB 00:00 2024-03-29T10:32:18.896 INFO:teuthology.orchestra.run.smithi086.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 389 kB/s | 49 kB 00:00 2024-03-29T10:32:18.913 INFO:teuthology.orchestra.run.smithi086.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-03-29T10:32:19.163 INFO:teuthology.orchestra.run.smithi086.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 768 kB/s | 192 kB 00:00 2024-03-29T10:32:19.222 INFO:teuthology.orchestra.run.smithi086.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.5 MB/s | 838 kB 00:00 2024-03-29T10:32:19.306 INFO:teuthology.orchestra.run.smithi086.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.2 MB/s | 4.4 MB 00:00 2024-03-29T10:32:19.514 INFO:teuthology.orchestra.run.smithi086.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 4.5 MB/s | 1.6 MB 00:00 2024-03-29T10:32:19.706 INFO:teuthology.orchestra.run.smithi086.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 7.8 MB/s | 3.1 MB 00:00 2024-03-29T10:32:19.764 INFO:teuthology.orchestra.run.smithi086.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 6.2 MB/s | 3.4 MB 00:00 2024-03-29T10:32:19.770 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:32:19.770 INFO:teuthology.orchestra.run.smithi086.stdout:Total 13 MB/s | 58 MB 00:04 2024-03-29T10:32:19.771 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T10:32:19.804 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T10:32:19.804 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T10:32:20.287 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T10:32:20.287 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T10:32:21.263 INFO:teuthology.orchestra.run.smithi084.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 2.5 MB/s | 20 MB 00:07 2024-03-29T10:32:21.268 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:32:21.268 INFO:teuthology.orchestra.run.smithi084.stdout:Total 7.3 MB/s | 58 MB 00:07 2024-03-29T10:32:21.268 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T10:32:21.306 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T10:32:21.306 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T10:32:21.735 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T10:32:21.766 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T10:32:21.767 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T10:32:21.809 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:32:21.845 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:32:21.870 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:32:21.893 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:32:21.909 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:32:22.051 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:32:22.106 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:32:22.492 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:32:22.538 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:32:22.630 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:32:22.720 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:32:22.740 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:32:22.802 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:32:22.831 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:32:22.867 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:32:22.936 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:32:22.958 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:32:23.014 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:32:23.049 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:32:23.168 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:32:23.200 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:32:23.223 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T10:32:23.241 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:32:23.273 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:32:23.302 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:32:23.322 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:32:23.335 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:32:23.383 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:32:23.493 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:32:23.515 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:32:23.533 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:32:23.655 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:32:23.675 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:32:23.681 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:32:23.729 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:32:23.763 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:32:23.795 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:32:23.835 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:32:24.052 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:32:24.119 INFO:teuthology.orchestra.run.smithi084.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:32:24.211 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:32:24.288 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:32:24.301 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:32:24.317 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:32:24.329 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:32:24.388 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:32:24.392 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:32:24.421 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:32:24.457 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:32:24.473 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:32:24.486 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:32:24.542 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:32:24.565 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:32:24.612 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:32:24.651 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:32:24.769 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:32:24.806 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:32:24.847 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:32:24.879 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:32:24.908 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:32:24.940 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:32:25.265 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:32:25.296 INFO:teuthology.orchestra.run.smithi084.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:32:25.368 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:32:25.400 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:32:25.468 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:32:25.917 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:32:25.948 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:32:26.003 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:32:26.070 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:32:26.100 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:32:26.702 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:32:26.749 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:32:27.297 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:32:27.336 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:32:27.433 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:32:27.433 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:32:27.433 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:32:27.433 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:32:27.470 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:32:28.232 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:32:28.272 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:32:28.818 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:32:28.858 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:32:28.949 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:32:28.949 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:32:28.949 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:32:28.949 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:32:28.984 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:32:36.969 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:32:36.969 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /sys 2024-03-29T10:32:36.969 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /proc 2024-03-29T10:32:36.969 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /mnt 2024-03-29T10:32:36.969 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /var/tmp 2024-03-29T10:32:36.969 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /home 2024-03-29T10:32:36.969 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /root 2024-03-29T10:32:36.969 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /tmp 2024-03-29T10:32:36.969 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:32:37.884 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:32:37.908 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:32:37.908 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:32:37.908 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:32:37.908 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:32:37.908 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:32:37.908 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:32:37.910 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:32:37.986 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:32:37.986 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:32:38.339 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:32:38.339 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /sys 2024-03-29T10:32:38.339 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /proc 2024-03-29T10:32:38.339 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /mnt 2024-03-29T10:32:38.339 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /var/tmp 2024-03-29T10:32:38.339 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /home 2024-03-29T10:32:38.339 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /root 2024-03-29T10:32:38.339 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /tmp 2024-03-29T10:32:38.339 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:32:39.078 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:32:39.078 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:32:39.078 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:32:39.078 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:32:39.078 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:32:39.078 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:32:39.078 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:32:39.078 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:32:39.079 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:32:39.079 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:32:39.079 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:32:39.079 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:32:39.079 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:32:39.079 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:32:39.079 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:32:39.079 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:32:39.080 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:32:39.080 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:32:39.080 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:32:39.080 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:32:39.080 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:32:39.080 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:32:39.080 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:32:39.080 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:32:39.080 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:32:39.080 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:32:39.080 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:32:39.080 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:32:39.080 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:32:39.080 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:32:39.081 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:32:39.081 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:32:39.081 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:32:39.081 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:32:39.265 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:32:39.290 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:32:39.290 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:32:39.290 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:32:39.290 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:32:39.290 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:32:39.290 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:32:39.292 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:32:39.359 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:32:39.359 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:32:40.169 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:32:40.169 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:32:40.169 INFO:teuthology.orchestra.run.smithi086.stdout:Upgraded: 2024-03-29T10:32:40.169 INFO:teuthology.orchestra.run.smithi086.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:40.169 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:40.169 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T10:32:40.169 INFO:teuthology.orchestra.run.smithi086.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:32:40.169 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:40.169 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:40.169 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:40.170 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:40.170 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:32:40.170 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:32:40.170 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:32:40.170 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:40.170 INFO:teuthology.orchestra.run.smithi086.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:32:40.170 INFO:teuthology.orchestra.run.smithi086.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:32:40.170 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:32:40.170 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:40.170 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:32:40.170 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:40.170 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:32:40.170 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:32:40.171 INFO:teuthology.orchestra.run.smithi086.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:32:40.171 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:40.171 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:40.171 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:40.171 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:32:40.171 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:40.171 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:40.171 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:40.171 INFO:teuthology.orchestra.run.smithi086.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:32:40.171 INFO:teuthology.orchestra.run.smithi086.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:32:40.171 INFO:teuthology.orchestra.run.smithi086.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:32:40.171 INFO:teuthology.orchestra.run.smithi086.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:32:40.171 INFO:teuthology.orchestra.run.smithi086.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:32:40.171 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:32:40.171 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T10:32:40.432 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:32:40.432 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:32:40.432 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:32:40.432 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:32:40.432 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:32:40.432 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:32:40.432 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:32:40.432 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:32:40.432 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:32:40.432 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:32:40.433 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:32:40.433 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:32:40.433 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:32:40.433 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:32:40.433 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:32:40.433 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:32:40.433 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:32:40.433 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:32:40.433 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:32:40.433 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:32:40.433 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:32:40.433 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:32:40.433 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:32:40.433 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:32:40.434 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:32:40.434 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:32:40.434 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:32:40.434 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:32:40.434 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:32:40.434 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:32:40.434 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:32:40.434 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:32:40.434 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:32:40.434 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:32:40.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-test 2024-03-29T10:32:41.100 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:29 ago on Fri 29 Mar 2024 10:32:12 AM UTC. 2024-03-29T10:32:41.261 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T10:32:41.261 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:32:41.261 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-29T10:32:41.262 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:32:41.262 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T10:32:41.262 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:32:41.262 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-29T10:32:41.262 INFO:teuthology.orchestra.run.smithi086.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:32:41.262 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:32:41.262 INFO:teuthology.orchestra.run.smithi086.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:32:41.262 INFO:teuthology.orchestra.run.smithi086.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:32:41.262 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:32:41.262 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:32:41.263 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T10:32:41.263 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:32:41.263 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2024-03-29T10:32:41.263 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:32:41.263 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 48 M 2024-03-29T10:32:41.264 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 204 M 2024-03-29T10:32:41.264 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T10:32:41.406 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:32:41.406 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:32:41.406 INFO:teuthology.orchestra.run.smithi084.stdout:Upgraded: 2024-03-29T10:32:41.406 INFO:teuthology.orchestra.run.smithi084.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:41.406 INFO:teuthology.orchestra.run.smithi084.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:41.406 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T10:32:41.406 INFO:teuthology.orchestra.run.smithi084.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:32:41.406 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:41.406 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:41.406 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:41.406 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:41.407 INFO:teuthology.orchestra.run.smithi084.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:32:41.407 INFO:teuthology.orchestra.run.smithi084.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:32:41.407 INFO:teuthology.orchestra.run.smithi084.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:32:41.407 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:41.407 INFO:teuthology.orchestra.run.smithi084.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:32:41.407 INFO:teuthology.orchestra.run.smithi084.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:32:41.407 INFO:teuthology.orchestra.run.smithi084.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:32:41.407 INFO:teuthology.orchestra.run.smithi084.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:41.407 INFO:teuthology.orchestra.run.smithi084.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:32:41.407 INFO:teuthology.orchestra.run.smithi084.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:41.407 INFO:teuthology.orchestra.run.smithi084.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:32:41.407 INFO:teuthology.orchestra.run.smithi084.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:32:41.407 INFO:teuthology.orchestra.run.smithi084.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:32:41.408 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:41.408 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:41.408 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:41.408 INFO:teuthology.orchestra.run.smithi084.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:32:41.408 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:41.408 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:41.408 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:41.408 INFO:teuthology.orchestra.run.smithi084.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:32:41.408 INFO:teuthology.orchestra.run.smithi084.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:32:41.408 INFO:teuthology.orchestra.run.smithi084.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:32:41.408 INFO:teuthology.orchestra.run.smithi084.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:32:41.408 INFO:teuthology.orchestra.run.smithi084.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:32:41.408 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:32:41.409 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T10:32:41.707 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-29T10:32:41.757 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-29T10:32:41.782 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 63 kB 00:00 2024-03-29T10:32:41.789 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-test 2024-03-29T10:32:41.924 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 421 kB/s | 163 kB 00:00 2024-03-29T10:32:42.343 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:32 ago on Fri 29 Mar 2024 10:32:10 AM UTC. 2024-03-29T10:32:42.492 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T10:32:42.493 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:32:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-29T10:32:42.493 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:32:42.493 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T10:32:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:32:42.493 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-29T10:32:42.493 INFO:teuthology.orchestra.run.smithi084.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:32:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:32:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:32:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:32:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:32:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:32:42.494 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T10:32:42.494 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:32:42.494 INFO:teuthology.orchestra.run.smithi084.stdout:Install 6 Packages 2024-03-29T10:32:42.494 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:32:42.495 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 48 M 2024-03-29T10:32:42.495 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 204 M 2024-03-29T10:32:42.495 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T10:32:42.674 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 210 kB/s | 187 kB 00:00 2024-03-29T10:32:42.957 INFO:teuthology.orchestra.run.smithi084.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-29T10:32:43.016 INFO:teuthology.orchestra.run.smithi084.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 300 kB 00:00 2024-03-29T10:32:43.050 INFO:teuthology.orchestra.run.smithi084.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T10:32:43.325 INFO:teuthology.orchestra.run.smithi084.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 291 kB/s | 163 kB 00:00 2024-03-29T10:32:43.883 INFO:teuthology.orchestra.run.smithi084.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 224 kB/s | 187 kB 00:00 2024-03-29T10:32:46.384 INFO:teuthology.orchestra.run.smithi084.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 13 MB/s | 47 MB 00:03 2024-03-29T10:32:46.387 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:32:46.387 INFO:teuthology.orchestra.run.smithi084.stdout:Total 12 MB/s | 48 MB 00:03 2024-03-29T10:32:46.403 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T10:32:46.424 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T10:32:46.425 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T10:32:46.802 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T10:32:46.803 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T10:32:47.377 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T10:32:47.458 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:32:47.501 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:32:47.541 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:32:47.593 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:32:47.620 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:32:48.433 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 6.9 MB/s | 47 MB 00:06 2024-03-29T10:32:48.435 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:32:48.436 INFO:teuthology.orchestra.run.smithi086.stdout:Total 6.7 MB/s | 48 MB 00:07 2024-03-29T10:32:48.449 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T10:32:48.469 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T10:32:48.469 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T10:32:48.851 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T10:32:48.852 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T10:32:49.424 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T10:32:49.510 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:32:49.548 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:32:49.580 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:32:49.632 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:32:49.661 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:32:52.085 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:32:52.169 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:32:54.116 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:32:54.210 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:32:54.581 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:32:54.581 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:32:54.581 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:32:54.582 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:32:54.582 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:32:54.582 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:32:54.930 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:32:54.931 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:32:54.931 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T10:32:54.931 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:54.931 INFO:teuthology.orchestra.run.smithi084.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:32:54.931 INFO:teuthology.orchestra.run.smithi084.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:54.931 INFO:teuthology.orchestra.run.smithi084.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:32:54.931 INFO:teuthology.orchestra.run.smithi084.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:32:54.931 INFO:teuthology.orchestra.run.smithi084.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:32:54.931 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:32:54.931 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T10:32:55.226 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph 2024-03-29T10:32:55.802 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:45 ago on Fri 29 Mar 2024 10:32:10 AM UTC. 2024-03-29T10:32:55.969 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T10:32:55.971 INFO:teuthology.orchestra.run.smithi084.stdout:========================================================================================= 2024-03-29T10:32:55.971 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-29T10:32:55.971 INFO:teuthology.orchestra.run.smithi084.stdout:========================================================================================= 2024-03-29T10:32:55.972 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T10:32:55.972 INFO:teuthology.orchestra.run.smithi084.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:32:55.972 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-29T10:32:55.972 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:32:55.972 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:32:55.972 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:32:55.972 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:32:55.972 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:32:55.972 INFO:teuthology.orchestra.run.smithi084.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:32:55.972 INFO:teuthology.orchestra.run.smithi084.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:32:55.972 INFO:teuthology.orchestra.run.smithi084.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:32:55.972 INFO:teuthology.orchestra.run.smithi084.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:32:55.972 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:32:55.973 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:32:55.973 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:32:55.973 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:32:55.973 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:32:55.973 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:32:55.973 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:32:55.973 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:32:55.973 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:32:55.973 INFO:teuthology.orchestra.run.smithi084.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:32:55.973 INFO:teuthology.orchestra.run.smithi084.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:32:55.973 INFO:teuthology.orchestra.run.smithi084.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:32:55.973 INFO:teuthology.orchestra.run.smithi084.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:32:55.973 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:32:55.973 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:32:55.973 INFO:teuthology.orchestra.run.smithi084.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:32:55.974 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:32:55.974 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:32:55.974 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:32:55.974 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:32:55.974 INFO:teuthology.orchestra.run.smithi084.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:32:55.974 INFO:teuthology.orchestra.run.smithi084.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:32:55.974 INFO:teuthology.orchestra.run.smithi084.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:32:55.974 INFO:teuthology.orchestra.run.smithi084.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:32:55.974 INFO:teuthology.orchestra.run.smithi084.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:32:55.974 INFO:teuthology.orchestra.run.smithi084.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:32:55.974 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:32:55.974 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T10:32:55.974 INFO:teuthology.orchestra.run.smithi084.stdout:========================================================================================= 2024-03-29T10:32:55.974 INFO:teuthology.orchestra.run.smithi084.stdout:Install 36 Packages 2024-03-29T10:32:55.975 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:32:55.975 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 30 M 2024-03-29T10:32:55.975 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 107 M 2024-03-29T10:32:55.975 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T10:32:56.553 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:32:56.554 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:32:56.554 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:32:56.554 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:32:56.554 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:32:56.554 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:32:56.842 INFO:teuthology.orchestra.run.smithi084.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 41 kB/s | 6.6 kB 00:00 2024-03-29T10:32:56.915 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:32:56.916 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:32:56.916 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T10:32:56.916 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:56.916 INFO:teuthology.orchestra.run.smithi086.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:32:56.916 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:32:56.916 INFO:teuthology.orchestra.run.smithi086.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:32:56.916 INFO:teuthology.orchestra.run.smithi086.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:32:56.916 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:32:56.916 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:32:56.916 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T10:32:57.092 INFO:teuthology.orchestra.run.smithi084.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 3.6 MB/s | 1.5 MB 00:00 2024-03-29T10:32:57.215 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph 2024-03-29T10:32:57.468 INFO:teuthology.orchestra.run.smithi084.stdout:(3/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 7.5 MB/s | 4.7 MB 00:00 2024-03-29T10:32:57.551 INFO:teuthology.orchestra.run.smithi084.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.9 MB/s | 247 kB 00:00 2024-03-29T10:32:57.626 INFO:teuthology.orchestra.run.smithi084.stdout:(5/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 2.2 MB/s | 2.1 MB 00:00 2024-03-29T10:32:57.693 INFO:teuthology.orchestra.run.smithi084.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-29T10:32:57.718 INFO:teuthology.orchestra.run.smithi084.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.6 MB/s | 144 kB 00:00 2024-03-29T10:32:57.769 INFO:teuthology.orchestra.run.smithi084.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 422 kB/s | 32 kB 00:00 2024-03-29T10:32:57.782 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:45 ago on Fri 29 Mar 2024 10:32:12 AM UTC. 2024-03-29T10:32:57.794 INFO:teuthology.orchestra.run.smithi084.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 628 kB/s | 47 kB 00:00 2024-03-29T10:32:57.819 INFO:teuthology.orchestra.run.smithi084.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 773 kB/s | 39 kB 00:00 2024-03-29T10:32:57.844 INFO:teuthology.orchestra.run.smithi084.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-29T10:32:57.895 INFO:teuthology.orchestra.run.smithi084.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-29T10:32:57.920 INFO:teuthology.orchestra.run.smithi084.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-29T10:32:57.934 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T10:32:57.937 INFO:teuthology.orchestra.run.smithi086.stdout:========================================================================================= 2024-03-29T10:32:57.937 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-29T10:32:57.937 INFO:teuthology.orchestra.run.smithi086.stdout:========================================================================================= 2024-03-29T10:32:57.937 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T10:32:57.937 INFO:teuthology.orchestra.run.smithi086.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:32:57.937 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-29T10:32:57.937 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:32:57.937 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:32:57.937 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:32:57.937 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:32:57.937 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:32:57.938 INFO:teuthology.orchestra.run.smithi086.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:32:57.938 INFO:teuthology.orchestra.run.smithi086.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:32:57.938 INFO:teuthology.orchestra.run.smithi086.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:32:57.938 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:32:57.938 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:32:57.938 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:32:57.938 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:32:57.938 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:32:57.938 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:32:57.938 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:32:57.938 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:32:57.938 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:32:57.938 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:32:57.939 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:32:57.939 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:32:57.939 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:32:57.939 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:32:57.939 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:32:57.939 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:32:57.939 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:32:57.939 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:32:57.939 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:32:57.939 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:32:57.939 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:32:57.939 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:32:57.939 INFO:teuthology.orchestra.run.smithi086.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:32:57.939 INFO:teuthology.orchestra.run.smithi086.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:32:57.940 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:32:57.940 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:32:57.940 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:32:57.940 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:32:57.940 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T10:32:57.940 INFO:teuthology.orchestra.run.smithi086.stdout:========================================================================================= 2024-03-29T10:32:57.940 INFO:teuthology.orchestra.run.smithi086.stdout:Install 36 Packages 2024-03-29T10:32:57.940 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:32:57.941 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 30 M 2024-03-29T10:32:57.941 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 107 M 2024-03-29T10:32:57.941 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T10:32:57.945 INFO:teuthology.orchestra.run.smithi084.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T10:32:57.971 INFO:teuthology.orchestra.run.smithi084.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-29T10:32:58.137 INFO:teuthology.orchestra.run.smithi084.stdout:(16/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 16 MB/s | 17 MB 00:01 2024-03-29T10:32:58.189 INFO:teuthology.orchestra.run.smithi084.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 103 kB/s | 22 kB 00:00 2024-03-29T10:32:58.329 INFO:teuthology.orchestra.run.smithi084.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 325 kB/s | 125 kB 00:00 2024-03-29T10:32:58.380 INFO:teuthology.orchestra.run.smithi084.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 260 kB/s | 43 kB 00:00 2024-03-29T10:32:58.386 INFO:teuthology.orchestra.run.smithi086.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 40 kB/s | 6.6 kB 00:00 2024-03-29T10:32:58.405 INFO:teuthology.orchestra.run.smithi084.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 565 kB/s | 151 kB 00:00 2024-03-29T10:32:58.464 INFO:teuthology.orchestra.run.smithi084.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 183 kB/s | 11 kB 00:00 2024-03-29T10:32:58.523 INFO:teuthology.orchestra.run.smithi084.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 302 kB/s | 18 kB 00:00 2024-03-29T10:32:58.547 INFO:teuthology.orchestra.run.smithi084.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.1 MB/s | 358 kB 00:00 2024-03-29T10:32:58.573 INFO:teuthology.orchestra.run.smithi084.stdout:(24/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 706 kB/s | 172 kB 00:00 2024-03-29T10:32:58.598 INFO:teuthology.orchestra.run.smithi084.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 307 kB/s | 23 kB 00:00 2024-03-29T10:32:58.623 INFO:teuthology.orchestra.run.smithi084.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 257 kB/s | 19 kB 00:00 2024-03-29T10:32:58.657 INFO:teuthology.orchestra.run.smithi084.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 235 kB/s | 20 kB 00:00 2024-03-29T10:32:58.682 INFO:teuthology.orchestra.run.smithi084.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 553 kB/s | 46 kB 00:00 2024-03-29T10:32:58.707 INFO:teuthology.orchestra.run.smithi084.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 941 kB/s | 79 kB 00:00 2024-03-29T10:32:58.758 INFO:teuthology.orchestra.run.smithi084.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 218 kB/s | 16 kB 00:00 2024-03-29T10:32:58.783 INFO:teuthology.orchestra.run.smithi084.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-03-29T10:32:58.803 INFO:teuthology.orchestra.run.smithi086.stdout:(2/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 11 MB/s | 4.7 MB 00:00 2024-03-29T10:32:58.808 INFO:teuthology.orchestra.run.smithi084.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 893 kB/s | 90 kB 00:00 2024-03-29T10:32:58.825 INFO:teuthology.orchestra.run.smithi084.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 533 kB/s | 36 kB 00:00 2024-03-29T10:32:58.859 INFO:teuthology.orchestra.run.smithi084.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-29T10:32:58.861 INFO:teuthology.orchestra.run.smithi086.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 3.3 MB/s | 2.1 MB 00:00 2024-03-29T10:32:58.884 INFO:teuthology.orchestra.run.smithi084.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 342 kB/s | 20 kB 00:00 2024-03-29T10:32:58.909 INFO:teuthology.orchestra.run.smithi084.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-03-29T10:32:58.912 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:32:58.912 INFO:teuthology.orchestra.run.smithi084.stdout:Total 10 MB/s | 30 MB 00:02 2024-03-29T10:32:58.934 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T10:32:58.970 INFO:teuthology.orchestra.run.smithi086.stdout:(4/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 2.0 MB/s | 1.5 MB 00:00 2024-03-29T10:32:58.972 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T10:32:58.972 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T10:32:58.995 INFO:teuthology.orchestra.run.smithi086.stdout:(5/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.8 MB/s | 247 kB 00:00 2024-03-29T10:32:59.154 INFO:teuthology.orchestra.run.smithi086.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-29T10:32:59.179 INFO:teuthology.orchestra.run.smithi086.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 903 kB/s | 144 kB 00:00 2024-03-29T10:32:59.196 INFO:teuthology.orchestra.run.smithi086.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 758 kB/s | 32 kB 00:00 2024-03-29T10:32:59.221 INFO:teuthology.orchestra.run.smithi086.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-29T10:32:59.246 INFO:teuthology.orchestra.run.smithi086.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 767 kB/s | 39 kB 00:00 2024-03-29T10:32:59.280 INFO:teuthology.orchestra.run.smithi086.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-29T10:32:59.339 INFO:teuthology.orchestra.run.smithi086.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 12 MB/s | 1.1 MB 00:00 2024-03-29T10:32:59.355 INFO:teuthology.orchestra.run.smithi086.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-29T10:32:59.376 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T10:32:59.376 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T10:32:59.380 INFO:teuthology.orchestra.run.smithi086.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-29T10:32:59.414 INFO:teuthology.orchestra.run.smithi086.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2024-03-29T10:32:59.665 INFO:teuthology.orchestra.run.smithi086.stdout:(16/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 19 MB/s | 17 MB 00:00 2024-03-29T10:32:59.707 INFO:teuthology.orchestra.run.smithi086.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 76 kB/s | 22 kB 00:00 2024-03-29T10:32:59.782 INFO:teuthology.orchestra.run.smithi086.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 312 kB/s | 125 kB 00:00 2024-03-29T10:33:00.074 INFO:teuthology.orchestra.run.smithi086.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 118 kB/s | 43 kB 00:00 2024-03-29T10:33:00.107 INFO:teuthology.orchestra.run.smithi086.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 342 kB/s | 151 kB 00:00 2024-03-29T10:33:00.243 INFO:teuthology.orchestra.run.smithi086.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 372 kB/s | 172 kB 00:00 2024-03-29T10:33:00.293 INFO:teuthology.orchestra.run.smithi086.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 57 kB/s | 11 kB 00:00 2024-03-29T10:33:00.511 INFO:teuthology.orchestra.run.smithi086.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 66 kB/s | 18 kB 00:00 2024-03-29T10:33:00.561 INFO:teuthology.orchestra.run.smithi086.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 87 kB/s | 23 kB 00:00 2024-03-29T10:33:00.586 INFO:teuthology.orchestra.run.smithi086.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 700 kB/s | 358 kB 00:00 2024-03-29T10:33:00.717 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T10:33:00.778 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:33:00.786 INFO:teuthology.orchestra.run.smithi086.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 71 kB/s | 19 kB 00:00 2024-03-29T10:33:00.826 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:33:00.828 INFO:teuthology.orchestra.run.smithi086.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 74 kB/s | 20 kB 00:00 2024-03-29T10:33:00.855 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:33:00.927 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:33:00.946 INFO:teuthology.orchestra.run.smithi086.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 129 kB/s | 46 kB 00:00 2024-03-29T10:33:00.971 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:33:01.004 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:33:01.104 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:33:01.128 INFO:teuthology.orchestra.run.smithi086.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 230 kB/s | 79 kB 00:00 2024-03-29T10:33:01.182 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:33:01.212 INFO:teuthology.orchestra.run.smithi086.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 62 kB/s | 16 kB 00:00 2024-03-29T10:33:01.223 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:33:01.295 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:33:01.349 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:33:01.386 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:33:01.387 INFO:teuthology.orchestra.run.smithi086.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 487 kB/s | 272 kB 00:00 2024-03-29T10:33:01.444 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:33:01.479 INFO:teuthology.orchestra.run.smithi086.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 134 kB/s | 36 kB 00:00 2024-03-29T10:33:01.500 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:33:01.563 INFO:teuthology.orchestra.run.smithi086.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 207 kB/s | 90 kB 00:00 2024-03-29T10:33:01.683 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:33:01.732 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:33:01.771 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:33:01.806 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:33:01.829 INFO:teuthology.orchestra.run.smithi086.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 75 kB/s | 20 kB 00:00 2024-03-29T10:33:01.835 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:33:01.912 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:33:01.930 INFO:teuthology.orchestra.run.smithi086.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 423 kB/s | 230 kB 00:00 2024-03-29T10:33:01.946 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:33:01.979 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:33:02.020 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:33:02.055 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:33:02.090 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:33:02.105 INFO:teuthology.orchestra.run.smithi086.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 683 kB/s | 427 kB 00:00 2024-03-29T10:33:02.110 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:02.110 INFO:teuthology.orchestra.run.smithi086.stdout:Total 7.1 MB/s | 30 MB 00:04 2024-03-29T10:33:02.135 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T10:33:02.177 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T10:33:02.177 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T10:33:02.179 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:33:02.220 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:33:02.292 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:33:02.428 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:33:02.555 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:33:02.601 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T10:33:02.602 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T10:33:02.710 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:33:02.733 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:33:02.758 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:33:02.759 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:33:02.759 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:33:02.759 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:33:02.759 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:33:02.759 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:03.953 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T10:33:03.999 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:33:04.033 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:33:04.059 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:33:04.130 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:33:04.192 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:33:04.200 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:33:04.218 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:33:04.219 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:33:04.219 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:33:04.219 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:33:04.219 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:33:04.219 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:04.233 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:33:04.325 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:33:04.412 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:33:04.453 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:33:04.525 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:33:04.578 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:33:04.616 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:33:04.659 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:33:04.664 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:33:04.682 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:33:04.682 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:33:04.683 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:33:04.683 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:33:04.683 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:33:04.683 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:04.721 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:33:04.893 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:33:04.906 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:33:04.916 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:33:04.916 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:33:04.916 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:33:04.916 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:33:04.916 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:33:04.916 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:04.953 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:33:04.982 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:33:04.983 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:33:05.027 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:33:05.082 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:33:05.158 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:33:05.192 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:33:05.225 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:33:05.265 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:33:05.300 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:33:05.335 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:33:05.424 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:33:05.457 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:33:05.538 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:33:05.676 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:33:05.800 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:33:05.956 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:33:05.987 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:33:06.012 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:33:06.013 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:33:06.013 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:33:06.013 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:33:06.013 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:33:06.013 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:06.688 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:33:06.689 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:33:06.689 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:33:06.689 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:33:06.689 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:33:06.689 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:33:06.689 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:33:06.689 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:33:06.689 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:33:06.689 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:33:06.689 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:33:06.689 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:33:06.689 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:33:06.690 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:33:06.690 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:33:06.690 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:33:06.690 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:33:06.690 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:33:06.690 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:33:06.690 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:33:06.690 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:33:06.690 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:33:06.690 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:33:06.690 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:33:06.690 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:33:06.690 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:33:06.692 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:33:06.692 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:33:06.692 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:33:06.692 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:33:06.692 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:33:06.692 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:33:06.692 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:33:06.692 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:33:06.692 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:33:06.692 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:33:07.454 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:33:07.481 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:33:07.481 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:33:07.481 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:33:07.481 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:33:07.481 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:33:07.481 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:07.644 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:33:07.645 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:07.645 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T10:33:07.645 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:07.645 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:07.645 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:07.645 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:07.645 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:07.645 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:07.645 INFO:teuthology.orchestra.run.smithi084.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:33:07.645 INFO:teuthology.orchestra.run.smithi084.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:33:07.645 INFO:teuthology.orchestra.run.smithi084.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:33:07.645 INFO:teuthology.orchestra.run.smithi084.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:33:07.646 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:33:07.646 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:33:07.646 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:33:07.646 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:33:07.646 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:33:07.646 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:33:07.646 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:33:07.646 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:33:07.646 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:33:07.646 INFO:teuthology.orchestra.run.smithi084.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:33:07.646 INFO:teuthology.orchestra.run.smithi084.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:33:07.646 INFO:teuthology.orchestra.run.smithi084.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:33:07.646 INFO:teuthology.orchestra.run.smithi084.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:33:07.647 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:33:07.647 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:33:07.647 INFO:teuthology.orchestra.run.smithi084.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:33:07.647 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:33:07.647 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:33:07.647 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:33:07.647 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:33:07.647 INFO:teuthology.orchestra.run.smithi084.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:33:07.647 INFO:teuthology.orchestra.run.smithi084.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:33:07.647 INFO:teuthology.orchestra.run.smithi084.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:33:07.647 INFO:teuthology.orchestra.run.smithi084.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:33:07.647 INFO:teuthology.orchestra.run.smithi084.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:33:07.647 INFO:teuthology.orchestra.run.smithi084.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:33:07.647 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:07.648 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T10:33:07.922 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:33:07.945 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:33:07.945 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:33:07.945 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:33:07.945 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:33:07.945 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:33:07.945 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:07.973 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-base 2024-03-29T10:33:08.155 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:33:08.178 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:33:08.179 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:33:08.179 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:33:08.179 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:33:08.179 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:33:08.179 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:08.244 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:33:08.543 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:58 ago on Fri 29 Mar 2024 10:32:10 AM UTC. 2024-03-29T10:33:08.641 INFO:teuthology.orchestra.run.smithi084.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:08.692 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T10:33:08.694 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-03-29T10:33:08.694 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T10:33:08.751 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install cephadm 2024-03-29T10:33:09.314 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 10:32:10 AM UTC. 2024-03-29T10:33:09.465 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T10:33:09.465 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:33:09.465 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-29T10:33:09.466 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:33:09.466 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T10:33:09.466 INFO:teuthology.orchestra.run.smithi084.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:33:09.466 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:09.466 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T10:33:09.466 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:33:09.466 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-29T10:33:09.466 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:09.466 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 747 k 2024-03-29T10:33:09.466 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 754 k 2024-03-29T10:33:09.466 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T10:33:09.748 INFO:teuthology.orchestra.run.smithi084.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.6 MB/s | 747 kB 00:00 2024-03-29T10:33:09.748 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:09.749 INFO:teuthology.orchestra.run.smithi084.stdout:Total 2.6 MB/s | 747 kB 00:00 2024-03-29T10:33:09.749 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T10:33:09.753 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T10:33:09.754 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T10:33:09.759 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T10:33:09.760 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T10:33:09.832 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T10:33:10.002 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:33:10.002 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:33:10.002 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:33:10.002 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:33:10.002 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:33:10.002 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:33:10.002 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:33:10.002 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:33:10.002 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:33:10.002 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:33:10.003 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:33:10.003 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:33:10.003 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:33:10.003 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:33:10.003 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:33:10.003 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:33:10.003 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:33:10.003 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:33:10.003 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:33:10.003 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:33:10.003 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:33:10.003 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:33:10.003 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:33:10.004 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:33:10.004 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:33:10.004 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:33:10.004 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:33:10.004 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:33:10.004 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:33:10.005 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:33:10.005 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:33:10.005 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:33:10.005 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:33:10.005 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:33:10.005 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:33:10.005 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:33:10.247 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:33:10.328 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:33:10.701 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:33:10.967 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:33:10.967 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:10.967 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T10:33:10.967 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:10.968 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:10.968 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:10.968 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:10.968 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:10.968 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:10.968 INFO:teuthology.orchestra.run.smithi086.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:33:10.968 INFO:teuthology.orchestra.run.smithi086.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:33:10.968 INFO:teuthology.orchestra.run.smithi086.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:33:10.968 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:33:10.968 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:33:10.968 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:33:10.968 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:33:10.968 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:33:10.968 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:33:10.969 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:33:10.969 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:33:10.969 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:33:10.969 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:33:10.969 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:33:10.969 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:33:10.969 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:33:10.969 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:33:10.969 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:33:10.969 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:33:10.969 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:33:10.969 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:33:10.969 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:33:10.969 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:33:10.969 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:33:10.970 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:33:10.970 INFO:teuthology.orchestra.run.smithi086.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:33:10.970 INFO:teuthology.orchestra.run.smithi086.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:33:10.970 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:33:10.970 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:33:10.970 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:33:10.970 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:10.970 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T10:33:11.014 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:33:11.014 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:11.014 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T10:33:11.014 INFO:teuthology.orchestra.run.smithi084.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:11.014 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:11.014 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T10:33:11.216 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:33:11.297 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-base 2024-03-29T10:33:11.789 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:01 ago on Fri 29 Mar 2024 10:32:10 AM UTC. 2024-03-29T10:33:11.864 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 10:32:12 AM UTC. 2024-03-29T10:33:11.942 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T10:33:11.943 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:33:11.943 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-29T10:33:11.943 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:33:11.943 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T10:33:11.943 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:33:11.943 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:11.943 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T10:33:11.943 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:33:11.944 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-29T10:33:11.944 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:11.944 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 145 k 2024-03-29T10:33:11.944 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 436 k 2024-03-29T10:33:11.944 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T10:33:11.962 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:12.014 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T10:33:12.015 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-29T10:33:12.015 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T10:33:12.083 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install cephadm 2024-03-29T10:33:12.303 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 405 kB/s | 145 kB 00:00 2024-03-29T10:33:12.303 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:12.303 INFO:teuthology.orchestra.run.smithi084.stdout:Total 404 kB/s | 145 kB 00:00 2024-03-29T10:33:12.303 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T10:33:12.312 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T10:33:12.312 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T10:33:12.366 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T10:33:12.366 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T10:33:12.505 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T10:33:12.570 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:33:12.596 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:33:12.596 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:33:12.596 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:33:12.596 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:12.643 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Mar 2024 10:32:12 AM UTC. 2024-03-29T10:33:12.794 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T10:33:12.795 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:33:12.795 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-29T10:33:12.795 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:33:12.795 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T10:33:12.795 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:33:12.795 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:12.795 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T10:33:12.795 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:33:12.796 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-29T10:33:12.796 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:12.796 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 747 k 2024-03-29T10:33:12.796 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 754 k 2024-03-29T10:33:12.796 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T10:33:13.234 INFO:teuthology.orchestra.run.smithi086.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.7 MB/s | 747 kB 00:00 2024-03-29T10:33:13.235 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:13.235 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.7 MB/s | 747 kB 00:00 2024-03-29T10:33:13.235 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T10:33:13.240 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T10:33:13.240 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T10:33:13.246 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T10:33:13.246 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T10:33:13.319 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T10:33:13.519 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:33:13.519 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:13.519 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T10:33:13.520 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:13.520 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:13.520 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T10:33:13.757 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr 2024-03-29T10:33:13.799 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:33:13.872 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:33:14.287 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:33:14.330 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:04 ago on Fri 29 Mar 2024 10:32:10 AM UTC. 2024-03-29T10:33:14.429 INFO:teuthology.orchestra.run.smithi084.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:14.482 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T10:33:14.482 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-03-29T10:33:14.482 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T10:33:14.545 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:33:14.584 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:33:14.584 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:14.584 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T10:33:14.585 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:14.585 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:14.585 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T10:33:14.771 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:33:15.106 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:05 ago on Fri 29 Mar 2024 10:32:10 AM UTC. 2024-03-29T10:33:15.256 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T10:33:15.257 INFO:teuthology.orchestra.run.smithi084.stdout:====================================================================================== 2024-03-29T10:33:15.257 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-29T10:33:15.257 INFO:teuthology.orchestra.run.smithi084.stdout:====================================================================================== 2024-03-29T10:33:15.257 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T10:33:15.257 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:33:15.257 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-29T10:33:15.257 INFO:teuthology.orchestra.run.smithi084.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:33:15.258 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:33:15.258 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:33:15.258 INFO:teuthology.orchestra.run.smithi084.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:33:15.258 INFO:teuthology.orchestra.run.smithi084.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:33:15.258 INFO:teuthology.orchestra.run.smithi084.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:33:15.258 INFO:teuthology.orchestra.run.smithi084.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:33:15.258 INFO:teuthology.orchestra.run.smithi084.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:33:15.258 INFO:teuthology.orchestra.run.smithi084.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:33:15.258 INFO:teuthology.orchestra.run.smithi084.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:33:15.258 INFO:teuthology.orchestra.run.smithi084.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:33:15.258 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:15.258 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T10:33:15.258 INFO:teuthology.orchestra.run.smithi084.stdout:====================================================================================== 2024-03-29T10:33:15.259 INFO:teuthology.orchestra.run.smithi084.stdout:Install 12 Packages 2024-03-29T10:33:15.259 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:15.259 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 6.9 M 2024-03-29T10:33:15.259 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 26 M 2024-03-29T10:33:15.259 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T10:33:15.337 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:03 ago on Fri 29 Mar 2024 10:32:12 AM UTC. 2024-03-29T10:33:15.490 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T10:33:15.491 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:33:15.491 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-29T10:33:15.491 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:33:15.491 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T10:33:15.491 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:33:15.491 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:15.491 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T10:33:15.492 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:33:15.492 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-29T10:33:15.492 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:15.492 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 145 k 2024-03-29T10:33:15.492 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 436 k 2024-03-29T10:33:15.492 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T10:33:15.806 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 463 kB/s | 145 kB 00:00 2024-03-29T10:33:15.807 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:15.807 INFO:teuthology.orchestra.run.smithi086.stdout:Total 461 kB/s | 145 kB 00:00 2024-03-29T10:33:15.807 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T10:33:15.814 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T10:33:15.814 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T10:33:15.866 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T10:33:15.866 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T10:33:15.931 INFO:teuthology.orchestra.run.smithi084.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 127 kB/s | 31 kB 00:00 2024-03-29T10:33:15.956 INFO:teuthology.orchestra.run.smithi084.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 56 kB/s | 15 kB 00:00 2024-03-29T10:33:16.006 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T10:33:16.082 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:33:16.098 INFO:teuthology.orchestra.run.smithi084.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.0 MB/s | 238 kB 00:00 2024-03-29T10:33:16.106 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:33:16.107 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:33:16.107 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:33:16.107 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:16.132 INFO:teuthology.orchestra.run.smithi084.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-03-29T10:33:16.182 INFO:teuthology.orchestra.run.smithi084.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.6 MB/s | 1.8 MB 00:00 2024-03-29T10:33:16.282 INFO:teuthology.orchestra.run.smithi084.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 253 kB/s | 19 kB 00:00 2024-03-29T10:33:16.324 INFO:teuthology.orchestra.run.smithi084.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.8 MB/s | 863 kB 00:00 2024-03-29T10:33:16.391 INFO:teuthology.orchestra.run.smithi084.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.1 MB/s | 548 kB 00:00 2024-03-29T10:33:16.433 INFO:teuthology.orchestra.run.smithi084.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 739 kB/s | 31 kB 00:00 2024-03-29T10:33:16.458 INFO:teuthology.orchestra.run.smithi084.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.1 MB/s | 144 kB 00:00 2024-03-29T10:33:16.483 INFO:teuthology.orchestra.run.smithi084.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-03-29T10:33:16.533 INFO:teuthology.orchestra.run.smithi084.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 8.1 MB/s | 2.0 MB 00:00 2024-03-29T10:33:16.537 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:16.537 INFO:teuthology.orchestra.run.smithi084.stdout:Total 5.4 MB/s | 6.9 MB 00:01 2024-03-29T10:33:16.555 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T10:33:16.572 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T10:33:16.572 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T10:33:16.736 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T10:33:16.737 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T10:33:17.093 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:33:17.093 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:17.093 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T10:33:17.093 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:17.093 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:17.093 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T10:33:17.254 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T10:33:17.333 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:33:17.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr 2024-03-29T10:33:17.377 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:33:17.423 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:33:17.454 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:33:17.561 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:33:17.679 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:33:17.774 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:33:17.817 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:33:17.845 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:33:17.877 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:33:17.939 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:05 ago on Fri 29 Mar 2024 10:32:12 AM UTC. 2024-03-29T10:33:18.038 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:18.090 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T10:33:18.091 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-29T10:33:18.091 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T10:33:18.121 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:33:18.163 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:33:18.176 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:33:18.724 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:06 ago on Fri 29 Mar 2024 10:32:12 AM UTC. 2024-03-29T10:33:18.880 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T10:33:18.881 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================== 2024-03-29T10:33:18.881 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-29T10:33:18.881 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================== 2024-03-29T10:33:18.882 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T10:33:18.882 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:33:18.882 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-29T10:33:18.882 INFO:teuthology.orchestra.run.smithi086.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:33:18.882 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:33:18.882 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:33:18.882 INFO:teuthology.orchestra.run.smithi086.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:33:18.882 INFO:teuthology.orchestra.run.smithi086.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:33:18.882 INFO:teuthology.orchestra.run.smithi086.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:33:18.883 INFO:teuthology.orchestra.run.smithi086.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:33:18.883 INFO:teuthology.orchestra.run.smithi086.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:33:18.883 INFO:teuthology.orchestra.run.smithi086.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:33:18.883 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:33:18.883 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:33:18.883 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:18.883 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T10:33:18.883 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================== 2024-03-29T10:33:18.883 INFO:teuthology.orchestra.run.smithi086.stdout:Install 12 Packages 2024-03-29T10:33:18.883 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:18.884 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 6.9 M 2024-03-29T10:33:18.884 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 26 M 2024-03-29T10:33:18.884 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T10:33:18.931 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:33:18.931 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:33:18.931 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:33:18.931 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:33:18.931 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:33:18.931 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:33:18.931 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:33:18.931 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:33:18.931 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:33:18.931 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:33:18.932 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:33:18.932 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:33:19.275 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:33:19.276 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:19.276 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T10:33:19.276 INFO:teuthology.orchestra.run.smithi084.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:33:19.276 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:19.276 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:19.277 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:19.277 INFO:teuthology.orchestra.run.smithi084.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:33:19.277 INFO:teuthology.orchestra.run.smithi084.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:33:19.277 INFO:teuthology.orchestra.run.smithi084.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:33:19.277 INFO:teuthology.orchestra.run.smithi084.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:33:19.277 INFO:teuthology.orchestra.run.smithi084.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:33:19.278 INFO:teuthology.orchestra.run.smithi084.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:33:19.278 INFO:teuthology.orchestra.run.smithi084.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:33:19.278 INFO:teuthology.orchestra.run.smithi084.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:33:19.278 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:19.278 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T10:33:19.501 INFO:teuthology.orchestra.run.smithi086.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 79 kB/s | 15 kB 00:00 2024-03-29T10:33:19.510 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:33:19.601 INFO:teuthology.orchestra.run.smithi086.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 107 kB/s | 31 kB 00:00 2024-03-29T10:33:19.668 INFO:teuthology.orchestra.run.smithi086.stdout:(3/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 4.9 MB/s | 1.8 MB 00:00 2024-03-29T10:33:19.710 INFO:teuthology.orchestra.run.smithi086.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-03-29T10:33:19.735 INFO:teuthology.orchestra.run.smithi086.stdout:(5/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 238 kB 00:00 2024-03-29T10:33:19.894 INFO:teuthology.orchestra.run.smithi086.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 123 kB/s | 19 kB 00:00 2024-03-29T10:33:19.961 INFO:teuthology.orchestra.run.smithi086.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.4 MB/s | 548 kB 00:00 2024-03-29T10:33:20.061 INFO:teuthology.orchestra.run.smithi086.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 12 MB/s | 2.0 MB 00:00 2024-03-29T10:33:20.070 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:10 ago on Fri 29 Mar 2024 10:32:10 AM UTC. 2024-03-29T10:33:20.095 INFO:teuthology.orchestra.run.smithi086.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.1 MB/s | 144 kB 00:00 2024-03-29T10:33:20.120 INFO:teuthology.orchestra.run.smithi086.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 528 kB/s | 31 kB 00:00 2024-03-29T10:33:20.145 INFO:teuthology.orchestra.run.smithi086.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.8 MB/s | 188 kB 00:00 2024-03-29T10:33:20.223 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T10:33:20.224 INFO:teuthology.orchestra.run.smithi084.stdout:============================================================================================ 2024-03-29T10:33:20.224 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-29T10:33:20.224 INFO:teuthology.orchestra.run.smithi084.stdout:============================================================================================ 2024-03-29T10:33:20.224 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T10:33:20.224 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:33:20.224 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-29T10:33:20.224 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:33:20.224 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:33:20.224 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:33:20.224 INFO:teuthology.orchestra.run.smithi084.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:33:20.224 INFO:teuthology.orchestra.run.smithi084.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:33:20.224 INFO:teuthology.orchestra.run.smithi084.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:33:20.224 INFO:teuthology.orchestra.run.smithi084.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:33:20.225 INFO:teuthology.orchestra.run.smithi084.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:33:20.225 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:33:20.225 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:33:20.225 INFO:teuthology.orchestra.run.smithi084.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:33:20.225 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:20.225 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T10:33:20.225 INFO:teuthology.orchestra.run.smithi084.stdout:============================================================================================ 2024-03-29T10:33:20.225 INFO:teuthology.orchestra.run.smithi084.stdout:Install 12 Packages 2024-03-29T10:33:20.225 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:20.226 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 37 M 2024-03-29T10:33:20.226 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 209 M 2024-03-29T10:33:20.226 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T10:33:20.681 INFO:teuthology.orchestra.run.smithi084.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 227 kB/s | 32 kB 00:00 2024-03-29T10:33:20.731 INFO:teuthology.orchestra.run.smithi084.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 351 kB/s | 17 kB 00:00 2024-03-29T10:33:20.765 INFO:teuthology.orchestra.run.smithi084.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-29T10:33:20.832 INFO:teuthology.orchestra.run.smithi084.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-03-29T10:33:21.124 INFO:teuthology.orchestra.run.smithi084.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 13 MB/s | 4.7 MB 00:00 2024-03-29T10:33:21.190 INFO:teuthology.orchestra.run.smithi084.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 4.8 MB 00:00 2024-03-29T10:33:21.216 INFO:teuthology.orchestra.run.smithi084.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-29T10:33:21.391 INFO:teuthology.orchestra.run.smithi084.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2024-03-29T10:33:21.441 INFO:teuthology.orchestra.run.smithi084.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 4.0 MB/s | 195 kB 00:00 2024-03-29T10:33:21.453 INFO:teuthology.orchestra.run.smithi086.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 484 kB/s | 863 kB 00:01 2024-03-29T10:33:21.460 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:21.460 INFO:teuthology.orchestra.run.smithi086.stdout:Total 2.7 MB/s | 6.9 MB 00:02 2024-03-29T10:33:21.480 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T10:33:21.499 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T10:33:21.499 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T10:33:21.608 INFO:teuthology.orchestra.run.smithi084.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-03-29T10:33:21.671 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T10:33:21.671 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T10:33:22.189 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T10:33:22.262 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:33:22.297 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:33:22.344 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:33:22.383 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:33:22.392 INFO:teuthology.orchestra.run.smithi084.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-1 4.0 MB/s | 7.4 MB 00:01 2024-03-29T10:33:22.467 INFO:teuthology.orchestra.run.smithi084.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 240 kB/s | 245 kB 00:01 2024-03-29T10:33:22.469 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:22.470 INFO:teuthology.orchestra.run.smithi084.stdout:Total 17 MB/s | 37 MB 00:02 2024-03-29T10:33:22.494 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:33:22.510 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T10:33:22.534 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T10:33:22.534 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T10:33:22.617 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:33:22.710 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:33:22.762 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:33:22.791 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:33:22.822 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:33:22.913 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T10:33:22.913 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T10:33:23.066 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:33:23.097 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:33:23.617 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T10:33:23.689 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:33:23.729 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:33:23.824 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:33:23.866 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:33:23.926 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:33:23.926 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:33:23.927 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:33:23.927 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:33:23.927 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:33:23.927 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:33:23.927 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:33:23.927 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:33:23.927 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:33:23.927 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:33:23.927 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:33:23.927 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:33:24.043 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:33:24.073 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:33:24.156 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:33:24.268 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:33:24.268 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:24.268 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T10:33:24.268 INFO:teuthology.orchestra.run.smithi086.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:33:24.268 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:24.268 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:24.268 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:24.268 INFO:teuthology.orchestra.run.smithi086.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:33:24.269 INFO:teuthology.orchestra.run.smithi086.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:33:24.269 INFO:teuthology.orchestra.run.smithi086.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:33:24.269 INFO:teuthology.orchestra.run.smithi086.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:33:24.269 INFO:teuthology.orchestra.run.smithi086.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:33:24.269 INFO:teuthology.orchestra.run.smithi086.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:33:24.269 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:33:24.269 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:33:24.269 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:24.269 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T10:33:24.525 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:33:24.810 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:33:24.927 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:33:25.092 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:13 ago on Fri 29 Mar 2024 10:32:12 AM UTC. 2024-03-29T10:33:25.258 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T10:33:25.258 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================ 2024-03-29T10:33:25.258 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-29T10:33:25.258 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================ 2024-03-29T10:33:25.258 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T10:33:25.258 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:33:25.258 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-29T10:33:25.258 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:33:25.258 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:33:25.259 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:33:25.259 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:33:25.259 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:33:25.259 INFO:teuthology.orchestra.run.smithi086.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:33:25.259 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:33:25.259 INFO:teuthology.orchestra.run.smithi086.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:33:25.259 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:33:25.259 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:33:25.259 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:33:25.259 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:25.259 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T10:33:25.259 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================ 2024-03-29T10:33:25.259 INFO:teuthology.orchestra.run.smithi086.stdout:Install 12 Packages 2024-03-29T10:33:25.259 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:25.260 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 37 M 2024-03-29T10:33:25.260 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 209 M 2024-03-29T10:33:25.260 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T10:33:25.785 INFO:teuthology.orchestra.run.smithi086.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 234 kB/s | 32 kB 00:00 2024-03-29T10:33:25.818 INFO:teuthology.orchestra.run.smithi086.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 540 kB/s | 17 kB 00:00 2024-03-29T10:33:25.860 INFO:teuthology.orchestra.run.smithi086.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 849 kB/s | 34 kB 00:00 2024-03-29T10:33:25.918 INFO:teuthology.orchestra.run.smithi086.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-29T10:33:26.094 INFO:teuthology.orchestra.run.smithi086.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-29T10:33:26.161 INFO:teuthology.orchestra.run.smithi086.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-29T10:33:26.172 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:33:26.194 INFO:teuthology.orchestra.run.smithi086.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-29T10:33:26.470 INFO:teuthology.orchestra.run.smithi086.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 51 MB/s | 16 MB 00:00 2024-03-29T10:33:26.562 INFO:teuthology.orchestra.run.smithi086.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-14 8.1 MB/s | 7.4 MB 00:00 2024-03-29T10:33:26.687 INFO:teuthology.orchestra.run.smithi086.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.9 MB/s | 245 kB 00:00 2024-03-29T10:33:27.012 INFO:teuthology.orchestra.run.smithi086.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 360 kB/s | 195 kB 00:00 2024-03-29T10:33:27.037 INFO:teuthology.orchestra.run.smithi086.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 956 kB/s | 803 kB 00:00 2024-03-29T10:33:27.039 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:27.039 INFO:teuthology.orchestra.run.smithi086.stdout:Total 21 MB/s | 37 MB 00:01 2024-03-29T10:33:27.080 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T10:33:27.103 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T10:33:27.103 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T10:33:27.210 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:33:27.238 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:33:27.471 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T10:33:27.472 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T10:33:28.173 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T10:33:28.235 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:33:28.275 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:33:28.367 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:33:28.403 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:33:28.581 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:33:28.610 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:33:28.690 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:33:29.342 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:33:29.455 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:33:29.777 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:33:29.778 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:33:29.778 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:33:29.778 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:33:29.778 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:33:29.778 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:33:29.778 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:33:29.778 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:33:29.778 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:33:29.778 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:33:29.778 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:33:29.778 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:33:30.194 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:33:30.194 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:30.194 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T10:33:30.194 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:30.194 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:33:30.194 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:33:30.194 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:33:30.195 INFO:teuthology.orchestra.run.smithi084.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:33:30.195 INFO:teuthology.orchestra.run.smithi084.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:33:30.195 INFO:teuthology.orchestra.run.smithi084.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:33:30.195 INFO:teuthology.orchestra.run.smithi084.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:33:30.195 INFO:teuthology.orchestra.run.smithi084.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:33:30.195 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:33:30.195 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:33:30.195 INFO:teuthology.orchestra.run.smithi084.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:33:30.195 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:30.195 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T10:33:30.506 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:33:30.705 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:33:31.076 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:21 ago on Fri 29 Mar 2024 10:32:10 AM UTC. 2024-03-29T10:33:31.232 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T10:33:31.233 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2024-03-29T10:33:31.234 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-29T10:33:31.234 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2024-03-29T10:33:31.234 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T10:33:31.234 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:33:31.234 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-29T10:33:31.234 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:33:31.234 INFO:teuthology.orchestra.run.smithi084.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:33:31.234 INFO:teuthology.orchestra.run.smithi084.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:33:31.234 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:33:31.234 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:33:31.234 INFO:teuthology.orchestra.run.smithi084.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:33:31.234 INFO:teuthology.orchestra.run.smithi084.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:33:31.235 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:33:31.235 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:33:31.235 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:33:31.235 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:33:31.235 INFO:teuthology.orchestra.run.smithi084.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:33:31.235 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:31.235 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T10:33:31.235 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2024-03-29T10:33:31.235 INFO:teuthology.orchestra.run.smithi084.stdout:Install 13 Packages 2024-03-29T10:33:31.235 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:31.236 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 2.0 M 2024-03-29T10:33:31.236 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 26 M 2024-03-29T10:33:31.236 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T10:33:31.405 INFO:teuthology.orchestra.run.smithi084.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 205 kB/s | 24 kB 00:00 2024-03-29T10:33:31.430 INFO:teuthology.orchestra.run.smithi084.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 126 kB/s | 18 kB 00:00 2024-03-29T10:33:31.481 INFO:teuthology.orchestra.run.smithi084.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T10:33:31.505 INFO:teuthology.orchestra.run.smithi084.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T10:33:31.531 INFO:teuthology.orchestra.run.smithi084.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T10:33:31.556 INFO:teuthology.orchestra.run.smithi084.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 854 kB/s | 43 kB 00:00 2024-03-29T10:33:31.581 INFO:teuthology.orchestra.run.smithi084.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 177 kB/s | 52 kB 00:00 2024-03-29T10:33:31.689 INFO:teuthology.orchestra.run.smithi084.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 112 kB/s | 15 kB 00:00 2024-03-29T10:33:31.715 INFO:teuthology.orchestra.run.smithi084.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 175 kB/s | 32 kB 00:00 2024-03-29T10:33:31.729 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:33:31.759 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:33:31.790 INFO:teuthology.orchestra.run.smithi084.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 971 kB/s | 203 kB 00:00 2024-03-29T10:33:31.824 INFO:teuthology.orchestra.run.smithi084.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 544 kB/s | 59 kB 00:00 2024-03-29T10:33:31.857 INFO:teuthology.orchestra.run.smithi084.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-03-29T10:33:31.949 INFO:teuthology.orchestra.run.smithi084.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.9 MB/s | 1.0 MB 00:00 2024-03-29T10:33:31.951 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:31.951 INFO:teuthology.orchestra.run.smithi084.stdout:Total 2.9 MB/s | 2.0 MB 00:00 2024-03-29T10:33:31.951 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T10:33:31.964 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T10:33:31.964 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T10:33:32.053 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T10:33:32.053 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T10:33:32.554 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T10:33:32.735 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:33:32.799 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:33:32.863 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:33:32.893 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:33:32.934 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:33:33.027 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:33:33.144 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:33:33.202 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:33:33.921 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:33:33.974 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:33:33.999 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:33:34.048 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:33:34.069 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:33:34.255 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:33:34.255 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:33:34.255 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:33:34.255 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:33:34.255 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:33:34.255 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:33:34.255 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:33:34.255 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:33:34.255 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:33:34.255 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:33:34.256 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:33:34.256 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:33:34.700 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:33:34.700 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:34.700 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T10:33:34.700 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:34.700 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:33:34.700 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:33:34.700 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:33:34.700 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:33:34.700 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:33:34.700 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:33:34.700 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:33:34.701 INFO:teuthology.orchestra.run.smithi086.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:33:34.701 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:33:34.701 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:33:34.701 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:33:34.701 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:34.701 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T10:33:34.928 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:33:34.928 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:33:34.928 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:33:34.928 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:33:34.928 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:33:34.928 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:33:34.929 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:33:34.929 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:33:34.929 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:33:34.929 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:33:34.929 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:33:34.929 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:33:34.929 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:33:35.061 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:33:35.291 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:33:35.291 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:35.291 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T10:33:35.291 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:35.291 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:33:35.291 INFO:teuthology.orchestra.run.smithi084.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:33:35.292 INFO:teuthology.orchestra.run.smithi084.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:33:35.292 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:33:35.292 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:33:35.292 INFO:teuthology.orchestra.run.smithi084.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:33:35.292 INFO:teuthology.orchestra.run.smithi084.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:33:35.292 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:33:35.292 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:33:35.292 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:33:35.292 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:33:35.293 INFO:teuthology.orchestra.run.smithi084.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:33:35.293 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:35.293 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T10:33:35.557 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:33:35.634 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:23 ago on Fri 29 Mar 2024 10:32:12 AM UTC. 2024-03-29T10:33:35.785 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T10:33:35.786 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================== 2024-03-29T10:33:35.787 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-29T10:33:35.787 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================== 2024-03-29T10:33:35.787 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T10:33:35.787 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:33:35.787 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-29T10:33:35.787 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:33:35.787 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:33:35.787 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:33:35.787 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:33:35.787 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:33:35.787 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:33:35.787 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:33:35.788 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:33:35.788 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:33:35.788 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:33:35.788 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:33:35.788 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:33:35.788 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:35.788 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T10:33:35.788 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================== 2024-03-29T10:33:35.789 INFO:teuthology.orchestra.run.smithi086.stdout:Install 13 Packages 2024-03-29T10:33:35.789 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:35.790 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 2.0 M 2024-03-29T10:33:35.790 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 26 M 2024-03-29T10:33:35.790 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T10:33:36.099 INFO:teuthology.orchestra.run.smithi086.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 201 kB/s | 24 kB 00:00 2024-03-29T10:33:36.116 INFO:teuthology.orchestra.run.smithi086.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 131 kB/s | 18 kB 00:00 2024-03-29T10:33:36.126 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:26 ago on Fri 29 Mar 2024 10:32:10 AM UTC. 2024-03-29T10:33:36.166 INFO:teuthology.orchestra.run.smithi086.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-29T10:33:36.192 INFO:teuthology.orchestra.run.smithi086.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T10:33:36.217 INFO:teuthology.orchestra.run.smithi086.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T10:33:36.242 INFO:teuthology.orchestra.run.smithi086.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 861 kB/s | 43 kB 00:00 2024-03-29T10:33:36.276 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T10:33:36.277 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2024-03-29T10:33:36.277 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-29T10:33:36.277 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2024-03-29T10:33:36.277 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T10:33:36.277 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:33:36.277 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-29T10:33:36.277 INFO:teuthology.orchestra.run.smithi084.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:33:36.277 INFO:teuthology.orchestra.run.smithi084.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:33:36.277 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:33:36.277 INFO:teuthology.orchestra.run.smithi084.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:33:36.277 INFO:teuthology.orchestra.run.smithi084.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:33:36.277 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:36.277 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T10:33:36.277 INFO:teuthology.orchestra.run.smithi084.stdout:======================================================================================== 2024-03-29T10:33:36.278 INFO:teuthology.orchestra.run.smithi084.stdout:Install 6 Packages 2024-03-29T10:33:36.278 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:36.278 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 6.8 M 2024-03-29T10:33:36.278 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 33 M 2024-03-29T10:33:36.278 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T10:33:36.442 INFO:teuthology.orchestra.run.smithi086.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 143 kB/s | 32 kB 00:00 2024-03-29T10:33:36.467 INFO:teuthology.orchestra.run.smithi086.stdout:(8/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 107 kB/s | 52 kB 00:00 2024-03-29T10:33:36.493 INFO:teuthology.orchestra.run.smithi086.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 60 kB/s | 15 kB 00:00 2024-03-29T10:33:36.868 INFO:teuthology.orchestra.run.smithi086.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 476 kB/s | 203 kB 00:00 2024-03-29T10:33:36.902 INFO:teuthology.orchestra.run.smithi086.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 145 kB/s | 59 kB 00:00 2024-03-29T10:33:37.191 INFO:teuthology.orchestra.run.smithi084.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 8.8 MB/s | 5.8 MB 00:00 2024-03-29T10:33:37.285 INFO:teuthology.orchestra.run.smithi086.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 215 kB/s | 90 kB 00:00 2024-03-29T10:33:37.783 INFO:teuthology.orchestra.run.smithi084.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 199 kB/s | 248 kB 00:01 2024-03-29T10:33:37.794 INFO:teuthology.orchestra.run.smithi086.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 788 kB/s | 1.0 MB 00:01 2024-03-29T10:33:37.796 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:37.797 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.0 MB/s | 2.0 MB 00:02 2024-03-29T10:33:37.797 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T10:33:37.800 INFO:teuthology.orchestra.run.smithi084.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 109 kB/s | 138 kB 00:01 2024-03-29T10:33:37.812 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T10:33:37.812 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T10:33:37.833 INFO:teuthology.orchestra.run.smithi084.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 854 kB/s | 548 kB 00:00 2024-03-29T10:33:37.908 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T10:33:37.908 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T10:33:38.159 INFO:teuthology.orchestra.run.smithi084.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 154 kB/s | 58 kB 00:00 2024-03-29T10:33:38.184 INFO:teuthology.orchestra.run.smithi084.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 133 kB/s | 51 kB 00:00 2024-03-29T10:33:38.187 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:38.188 INFO:teuthology.orchestra.run.smithi084.stdout:Total 3.6 MB/s | 6.8 MB 00:01 2024-03-29T10:33:38.201 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T10:33:38.211 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T10:33:38.211 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T10:33:38.302 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T10:33:38.303 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T10:33:38.454 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T10:33:38.592 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T10:33:38.623 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:33:38.677 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:33:38.741 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:33:38.758 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:33:38.770 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:33:38.802 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:33:38.811 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:33:38.913 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:33:39.021 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:33:39.071 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:33:39.327 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:33:39.415 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:33:39.475 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:33:39.499 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:33:39.794 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:33:39.844 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:33:39.886 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:33:39.934 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:33:39.956 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:33:40.282 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:33:40.282 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:33:40.282 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:33:40.282 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:33:40.282 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:33:40.282 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:33:40.589 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:33:40.589 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:40.589 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T10:33:40.589 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:40.589 INFO:teuthology.orchestra.run.smithi084.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:33:40.589 INFO:teuthology.orchestra.run.smithi084.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:33:40.589 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:33:40.589 INFO:teuthology.orchestra.run.smithi084.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:33:40.589 INFO:teuthology.orchestra.run.smithi084.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:33:40.590 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:40.590 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T10:33:40.758 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:33:40.758 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:33:40.758 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:33:40.758 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:33:40.758 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:33:40.758 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:33:40.758 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:33:40.758 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:33:40.758 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:33:40.758 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:33:40.758 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:33:40.759 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:33:40.759 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:33:40.818 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-fuse 2024-03-29T10:33:41.114 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:33:41.114 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:41.114 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T10:33:41.114 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:41.114 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:33:41.114 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:33:41.114 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:33:41.114 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:33:41.114 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:33:41.114 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:33:41.114 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:33:41.115 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:33:41.115 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:33:41.115 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:33:41.115 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:33:41.115 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:33:41.115 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:41.115 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T10:33:41.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:33:41.381 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:31 ago on Fri 29 Mar 2024 10:32:10 AM UTC. 2024-03-29T10:33:41.531 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T10:33:41.531 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:33:41.531 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-29T10:33:41.531 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:33:41.531 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T10:33:41.531 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:33:41.531 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-29T10:33:41.531 INFO:teuthology.orchestra.run.smithi084.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:33:41.532 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:41.532 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T10:33:41.532 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:33:41.532 INFO:teuthology.orchestra.run.smithi084.stdout:Install 2 Packages 2024-03-29T10:33:41.532 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:41.532 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 964 k 2024-03-29T10:33:41.532 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 2.8 M 2024-03-29T10:33:41.532 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T10:33:41.728 INFO:teuthology.orchestra.run.smithi084.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 402 kB/s | 79 kB 00:00 2024-03-29T10:33:41.919 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:29 ago on Fri 29 Mar 2024 10:32:12 AM UTC. 2024-03-29T10:33:42.071 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T10:33:42.071 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================== 2024-03-29T10:33:42.071 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-29T10:33:42.072 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================== 2024-03-29T10:33:42.072 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T10:33:42.072 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:33:42.072 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-29T10:33:42.072 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:33:42.072 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:33:42.072 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:33:42.072 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:33:42.072 INFO:teuthology.orchestra.run.smithi086.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:33:42.072 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:42.072 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T10:33:42.072 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================== 2024-03-29T10:33:42.073 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2024-03-29T10:33:42.073 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:42.073 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 6.8 M 2024-03-29T10:33:42.074 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 33 M 2024-03-29T10:33:42.074 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T10:33:42.720 INFO:teuthology.orchestra.run.smithi084.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 746 kB/s | 885 kB 00:01 2024-03-29T10:33:42.721 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:42.721 INFO:teuthology.orchestra.run.smithi084.stdout:Total 810 kB/s | 964 kB 00:01 2024-03-29T10:33:42.722 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T10:33:42.742 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T10:33:42.742 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T10:33:42.815 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T10:33:42.815 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T10:33:42.820 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 5.8 MB 00:00 2024-03-29T10:33:42.921 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 253 kB/s | 138 kB 00:00 2024-03-29T10:33:42.997 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T10:33:43.145 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:33:43.187 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 216 kB/s | 58 kB 00:00 2024-03-29T10:33:43.220 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:33:43.229 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.3 MB/s | 548 kB 00:00 2024-03-29T10:33:43.254 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 767 kB/s | 51 kB 00:00 2024-03-29T10:33:43.355 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 253 kB/s | 248 kB 00:00 2024-03-29T10:33:43.358 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:43.358 INFO:teuthology.orchestra.run.smithi086.stdout:Total 5.3 MB/s | 6.8 MB 00:01 2024-03-29T10:33:43.371 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T10:33:43.380 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T10:33:43.381 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T10:33:43.467 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T10:33:43.468 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T10:33:43.755 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T10:33:43.894 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:33:43.894 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:33:43.929 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:33:43.981 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:33:44.181 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:33:44.181 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:44.181 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T10:33:44.181 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:33:44.181 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:44.181 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T10:33:44.438 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-volume 2024-03-29T10:33:44.508 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:33:44.592 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:33:44.653 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:33:44.676 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:33:45.014 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 10:32:10 AM UTC. 2024-03-29T10:33:45.170 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T10:33:45.171 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:33:45.171 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-03-29T10:33:45.171 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:33:45.171 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T10:33:45.171 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:33:45.171 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-03-29T10:33:45.171 INFO:teuthology.orchestra.run.smithi084.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:33:45.171 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:45.171 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T10:33:45.172 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:33:45.172 INFO:teuthology.orchestra.run.smithi084.stdout:Install 2 Packages 2024-03-29T10:33:45.172 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:45.172 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 332 k 2024-03-29T10:33:45.172 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 1.5 M 2024-03-29T10:33:45.172 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T10:33:45.307 INFO:teuthology.orchestra.run.smithi084.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 508 kB/s | 69 kB 00:00 2024-03-29T10:33:45.463 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:33:45.463 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:33:45.463 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:33:45.463 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:33:45.463 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:33:45.463 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:33:45.641 INFO:teuthology.orchestra.run.smithi084.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 562 kB/s | 264 kB 00:00 2024-03-29T10:33:45.642 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:45.642 INFO:teuthology.orchestra.run.smithi084.stdout:Total 706 kB/s | 332 kB 00:00 2024-03-29T10:33:45.642 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T10:33:45.651 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T10:33:45.651 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T10:33:45.690 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T10:33:45.690 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T10:33:45.763 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:33:45.763 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:45.763 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T10:33:45.763 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:45.763 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:33:45.763 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:33:45.763 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:33:45.763 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:33:45.763 INFO:teuthology.orchestra.run.smithi086.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:33:45.764 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:45.764 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T10:33:45.835 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T10:33:46.004 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-fuse 2024-03-29T10:33:46.015 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:33:46.060 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:33:46.076 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:33:46.076 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:33:46.077 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:33:46.077 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:46.566 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:34 ago on Fri 29 Mar 2024 10:32:12 AM UTC. 2024-03-29T10:33:46.717 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T10:33:46.718 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:33:46.718 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-29T10:33:46.718 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:33:46.718 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T10:33:46.718 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:33:46.718 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-29T10:33:46.718 INFO:teuthology.orchestra.run.smithi086.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:33:46.718 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:46.718 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T10:33:46.718 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:33:46.718 INFO:teuthology.orchestra.run.smithi086.stdout:Install 2 Packages 2024-03-29T10:33:46.719 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:46.719 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 964 k 2024-03-29T10:33:46.719 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 2.8 M 2024-03-29T10:33:46.719 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T10:33:46.760 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:33:46.881 INFO:teuthology.orchestra.run.smithi086.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 488 kB/s | 79 kB 00:00 2024-03-29T10:33:47.099 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:33:47.099 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:47.099 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T10:33:47.099 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:47.099 INFO:teuthology.orchestra.run.smithi084.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:33:47.099 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:47.099 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T10:33:47.471 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librados-devel 2024-03-29T10:33:47.697 INFO:teuthology.orchestra.run.smithi086.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 905 kB/s | 885 kB 00:00 2024-03-29T10:33:47.699 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:47.699 INFO:teuthology.orchestra.run.smithi086.stdout:Total 983 kB/s | 964 kB 00:00 2024-03-29T10:33:47.700 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T10:33:47.723 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T10:33:47.723 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T10:33:47.800 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T10:33:47.800 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T10:33:47.989 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T10:33:48.047 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 10:32:10 AM UTC. 2024-03-29T10:33:48.132 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:33:48.197 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T10:33:48.198 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:33:48.198 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-29T10:33:48.198 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:33:48.198 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T10:33:48.198 INFO:teuthology.orchestra.run.smithi084.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:33:48.198 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:48.199 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T10:33:48.199 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:33:48.199 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-29T10:33:48.199 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:48.199 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 127 k 2024-03-29T10:33:48.199 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 456 k 2024-03-29T10:33:48.199 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T10:33:48.207 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:33:48.479 INFO:teuthology.orchestra.run.smithi084.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 453 kB/s | 127 kB 00:00 2024-03-29T10:33:48.480 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:48.480 INFO:teuthology.orchestra.run.smithi084.stdout:Total 450 kB/s | 127 kB 00:00 2024-03-29T10:33:48.480 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T10:33:48.485 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T10:33:48.485 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T10:33:48.541 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T10:33:48.541 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T10:33:48.685 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T10:33:48.778 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:48.912 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:33:48.912 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:33:49.234 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:49.280 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:33:49.280 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:49.280 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T10:33:49.280 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:33:49.280 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:49.280 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T10:33:49.516 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:49.516 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:49.516 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T10:33:49.516 INFO:teuthology.orchestra.run.smithi084.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:49.516 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:49.517 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T10:33:49.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-volume 2024-03-29T10:33:49.723 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install libcephfs2 2024-03-29T10:33:50.161 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 10:32:12 AM UTC. 2024-03-29T10:33:50.308 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 10:32:10 AM UTC. 2024-03-29T10:33:50.313 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T10:33:50.314 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:33:50.314 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-29T10:33:50.314 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:33:50.314 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T10:33:50.314 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:33:50.314 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-29T10:33:50.314 INFO:teuthology.orchestra.run.smithi086.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:33:50.314 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:50.314 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T10:33:50.314 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:33:50.314 INFO:teuthology.orchestra.run.smithi086.stdout:Install 2 Packages 2024-03-29T10:33:50.315 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:50.315 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 332 k 2024-03-29T10:33:50.315 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 1.5 M 2024-03-29T10:33:50.315 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T10:33:50.407 INFO:teuthology.orchestra.run.smithi084.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:50.460 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T10:33:50.461 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-03-29T10:33:50.461 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T10:33:50.527 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install libcephfs-devel 2024-03-29T10:33:50.568 INFO:teuthology.orchestra.run.smithi086.stdout:(1/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 1.0 MB/s | 264 kB 00:00 2024-03-29T10:33:50.594 INFO:teuthology.orchestra.run.smithi086.stdout:(2/2): python3-packaging-20.9-5.el9.noarch.rpm 247 kB/s | 69 kB 00:00 2024-03-29T10:33:50.595 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:50.595 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-29T10:33:50.595 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T10:33:50.604 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T10:33:50.604 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T10:33:50.639 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T10:33:50.640 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T10:33:50.790 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T10:33:50.970 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:33:51.004 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:33:51.022 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:33:51.022 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:33:51.022 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:33:51.022 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:51.085 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:32:10 AM UTC. 2024-03-29T10:33:51.235 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T10:33:51.236 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:33:51.236 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-29T10:33:51.236 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:33:51.236 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T10:33:51.237 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:33:51.237 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:51.237 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T10:33:51.237 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:33:51.237 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-29T10:33:51.237 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:51.237 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 31 k 2024-03-29T10:33:51.237 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 140 k 2024-03-29T10:33:51.237 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T10:33:51.598 INFO:teuthology.orchestra.run.smithi084.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 86 kB/s | 31 kB 00:00 2024-03-29T10:33:51.599 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:51.599 INFO:teuthology.orchestra.run.smithi084.stdout:Total 86 kB/s | 31 kB 00:00 2024-03-29T10:33:51.599 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T10:33:51.602 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T10:33:51.603 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T10:33:51.629 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T10:33:51.629 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T10:33:51.748 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:33:51.757 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T10:33:51.855 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:52.097 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:33:52.097 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:52.097 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T10:33:52.097 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:33:52.097 INFO:teuthology.orchestra.run.smithi086.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:33:52.097 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:52.097 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T10:33:52.221 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:52.474 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:52.475 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:52.475 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T10:33:52.475 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:52.475 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:52.475 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T10:33:52.483 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados-devel 2024-03-29T10:33:52.611 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librados2 2024-03-29T10:33:53.048 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 10:32:12 AM UTC. 2024-03-29T10:33:53.172 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 10:32:10 AM UTC. 2024-03-29T10:33:53.200 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T10:33:53.201 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:33:53.201 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-29T10:33:53.201 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:33:53.201 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T10:33:53.201 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:33:53.201 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:53.201 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T10:33:53.201 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:33:53.201 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-29T10:33:53.201 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:53.201 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 127 k 2024-03-29T10:33:53.201 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 456 k 2024-03-29T10:33:53.202 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T10:33:53.268 INFO:teuthology.orchestra.run.smithi084.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:53.319 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T10:33:53.320 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-03-29T10:33:53.320 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T10:33:53.368 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librbd1 2024-03-29T10:33:53.524 INFO:teuthology.orchestra.run.smithi086.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 394 kB/s | 127 kB 00:00 2024-03-29T10:33:53.524 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:53.524 INFO:teuthology.orchestra.run.smithi086.stdout:Total 392 kB/s | 127 kB 00:00 2024-03-29T10:33:53.524 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T10:33:53.530 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T10:33:53.530 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T10:33:53.585 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T10:33:53.585 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T10:33:53.724 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T10:33:53.813 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:53.924 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 10:32:10 AM UTC. 2024-03-29T10:33:54.020 INFO:teuthology.orchestra.run.smithi084.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:54.072 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T10:33:54.073 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-03-29T10:33:54.073 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T10:33:54.114 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rados 2024-03-29T10:33:54.268 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:54.544 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:54.544 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:54.544 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T10:33:54.544 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:54.544 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:54.545 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T10:33:54.679 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 10:32:10 AM UTC. 2024-03-29T10:33:54.778 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:54.815 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs2 2024-03-29T10:33:54.830 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T10:33:54.831 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-03-29T10:33:54.832 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T10:33:54.878 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rgw 2024-03-29T10:33:55.422 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 10:32:12 AM UTC. 2024-03-29T10:33:55.446 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 10:32:10 AM UTC. 2024-03-29T10:33:55.527 INFO:teuthology.orchestra.run.smithi086.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:55.545 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:55.582 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T10:33:55.583 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-29T10:33:55.583 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T10:33:55.596 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T10:33:55.598 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-03-29T10:33:55.598 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T10:33:55.641 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-cephfs 2024-03-29T10:33:55.670 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs-devel 2024-03-29T10:33:56.209 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 10:32:10 AM UTC. 2024-03-29T10:33:56.255 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 10:32:12 AM UTC. 2024-03-29T10:33:56.307 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:56.360 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T10:33:56.361 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-03-29T10:33:56.361 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T10:33:56.407 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rbd 2024-03-29T10:33:56.408 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T10:33:56.408 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:33:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-29T10:33:56.409 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:33:56.409 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T10:33:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:33:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:56.409 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T10:33:56.409 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:33:56.409 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-29T10:33:56.409 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:56.409 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 31 k 2024-03-29T10:33:56.409 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 140 k 2024-03-29T10:33:56.409 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T10:33:56.670 INFO:teuthology.orchestra.run.smithi086.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 119 kB/s | 31 kB 00:00 2024-03-29T10:33:56.671 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:56.671 INFO:teuthology.orchestra.run.smithi086.stdout:Total 119 kB/s | 31 kB 00:00 2024-03-29T10:33:56.671 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T10:33:56.675 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T10:33:56.675 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T10:33:56.705 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T10:33:56.705 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T10:33:56.849 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T10:33:56.952 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:56.969 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 10:32:10 AM UTC. 2024-03-29T10:33:57.067 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:57.119 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T10:33:57.120 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-03-29T10:33:57.120 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T10:33:57.169 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-fuse 2024-03-29T10:33:57.326 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:57.577 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:57.577 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:57.577 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T10:33:57.578 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:57.578 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:33:57.578 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T10:33:57.744 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 10:32:10 AM UTC. 2024-03-29T10:33:57.751 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados2 2024-03-29T10:33:57.904 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T10:33:57.905 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:33:57.905 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-03-29T10:33:57.905 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:33:57.905 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T10:33:57.905 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:33:57.905 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:57.905 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T10:33:57.905 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:33:57.905 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-29T10:33:57.906 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:57.906 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 85 k 2024-03-29T10:33:57.906 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 231 k 2024-03-29T10:33:57.906 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T10:33:58.184 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 307 kB/s | 85 kB 00:00 2024-03-29T10:33:58.184 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:33:58.185 INFO:teuthology.orchestra.run.smithi084.stdout:Total 305 kB/s | 85 kB 00:00 2024-03-29T10:33:58.185 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T10:33:58.192 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T10:33:58.192 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T10:33:58.248 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T10:33:58.248 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T10:33:58.310 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 10:32:12 AM UTC. 2024-03-29T10:33:58.383 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T10:33:58.406 INFO:teuthology.orchestra.run.smithi086.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:58.457 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T10:33:58.458 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-29T10:33:58.458 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T10:33:58.491 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:58.517 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librbd1 2024-03-29T10:33:58.888 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:59.074 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 10:32:12 AM UTC. 2024-03-29T10:33:59.159 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:33:59.159 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:59.159 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T10:33:59.159 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:33:59.159 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:33:59.159 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T10:33:59.171 INFO:teuthology.orchestra.run.smithi086.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:59.222 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T10:33:59.223 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-29T10:33:59.223 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T10:33:59.275 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rados 2024-03-29T10:33:59.385 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-mirror 2024-03-29T10:33:59.838 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 10:32:12 AM UTC. 2024-03-29T10:33:59.937 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:33:59.966 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 10:32:10 AM UTC. 2024-03-29T10:33:59.988 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T10:33:59.989 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-29T10:33:59.990 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T10:34:00.039 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rgw 2024-03-29T10:34:00.124 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T10:34:00.125 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:34:00.125 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-03-29T10:34:00.125 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:34:00.125 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T10:34:00.125 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:34:00.125 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:00.125 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T10:34:00.125 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:34:00.125 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-29T10:34:00.125 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:00.125 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 3.1 M 2024-03-29T10:34:00.126 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 13 M 2024-03-29T10:34:00.126 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T10:34:00.597 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 10:32:12 AM UTC. 2024-03-29T10:34:00.695 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:34:00.746 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T10:34:00.747 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-29T10:34:00.747 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T10:34:00.800 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-cephfs 2024-03-29T10:34:01.178 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 2.9 MB/s | 3.1 MB 00:01 2024-03-29T10:34:01.179 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:34:01.179 INFO:teuthology.orchestra.run.smithi084.stdout:Total 2.9 MB/s | 3.1 MB 00:01 2024-03-29T10:34:01.179 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T10:34:01.188 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T10:34:01.188 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T10:34:01.262 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T10:34:01.263 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T10:34:01.368 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 10:32:12 AM UTC. 2024-03-29T10:34:01.467 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:34:01.518 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T10:34:01.520 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-29T10:34:01.520 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T10:34:01.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rbd 2024-03-29T10:34:01.687 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T10:34:01.753 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:34:01.779 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:34:01.779 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:34:01.779 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:34:01.779 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:34:01.779 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:34:01.779 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:02.129 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:50 ago on Fri 29 Mar 2024 10:32:12 AM UTC. 2024-03-29T10:34:02.228 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:34:02.280 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T10:34:02.281 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-29T10:34:02.281 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T10:34:02.330 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-fuse 2024-03-29T10:34:02.887 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:50 ago on Fri 29 Mar 2024 10:32:12 AM UTC. 2024-03-29T10:34:02.900 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:34:02.900 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:02.901 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T10:34:02.901 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:02.901 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:02.901 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T10:34:03.034 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T10:34:03.035 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:34:03.035 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-03-29T10:34:03.035 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:34:03.035 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T10:34:03.035 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:34:03.035 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:03.035 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T10:34:03.035 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:34:03.035 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-29T10:34:03.035 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:03.035 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 85 k 2024-03-29T10:34:03.036 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 231 k 2024-03-29T10:34:03.036 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T10:34:03.156 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-nbd 2024-03-29T10:34:03.257 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 386 kB/s | 85 kB 00:00 2024-03-29T10:34:03.257 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:34:03.257 INFO:teuthology.orchestra.run.smithi086.stdout:Total 384 kB/s | 85 kB 00:00 2024-03-29T10:34:03.257 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T10:34:03.264 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T10:34:03.264 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T10:34:03.321 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T10:34:03.321 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T10:34:03.459 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T10:34:03.546 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:34:03.726 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:53 ago on Fri 29 Mar 2024 10:32:10 AM UTC. 2024-03-29T10:34:03.876 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-03-29T10:34:03.877 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:34:03.877 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-03-29T10:34:03.877 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:34:03.877 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-03-29T10:34:03.877 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:34:03.877 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:03.878 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-03-29T10:34:03.878 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-03-29T10:34:03.878 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-03-29T10:34:03.878 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:03.878 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 171 k 2024-03-29T10:34:03.878 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 494 k 2024-03-29T10:34:03.878 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-03-29T10:34:03.938 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:34:04.222 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 499 kB/s | 171 kB 00:00 2024-03-29T10:34:04.223 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:34:04.223 INFO:teuthology.orchestra.run.smithi084.stdout:Total 497 kB/s | 171 kB 00:00 2024-03-29T10:34:04.223 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-03-29T10:34:04.230 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:34:04.230 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:04.230 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T10:34:04.230 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:04.230 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:04.230 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T10:34:04.231 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-03-29T10:34:04.231 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-03-29T10:34:04.287 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-03-29T10:34:04.288 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-03-29T10:34:04.433 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-03-29T10:34:04.457 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-mirror 2024-03-29T10:34:04.529 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:34:04.937 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:34:05.025 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:53 ago on Fri 29 Mar 2024 10:32:12 AM UTC. 2024-03-29T10:34:05.181 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T10:34:05.181 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:34:05.181 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-29T10:34:05.182 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:34:05.182 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T10:34:05.182 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:34:05.182 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:05.182 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T10:34:05.182 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:34:05.182 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-29T10:34:05.182 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:05.182 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 3.1 M 2024-03-29T10:34:05.183 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 13 M 2024-03-29T10:34:05.183 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T10:34:05.240 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:34:05.241 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:05.241 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-03-29T10:34:05.241 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:05.241 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:05.241 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-03-29T10:34:05.466 DEBUG:teuthology.parallel:result is None 2024-03-29T10:34:05.801 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 5.0 MB/s | 3.1 MB 00:00 2024-03-29T10:34:05.801 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:34:05.801 INFO:teuthology.orchestra.run.smithi086.stdout:Total 5.0 MB/s | 3.1 MB 00:00 2024-03-29T10:34:05.802 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T10:34:05.811 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T10:34:05.811 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T10:34:05.885 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T10:34:05.885 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T10:34:06.316 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T10:34:06.386 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:34:06.409 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:34:06.409 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:34:06.409 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:34:06.409 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:34:06.409 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:34:06.409 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:07.620 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:34:07.620 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:07.620 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T10:34:07.620 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:07.620 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:07.620 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T10:34:07.905 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-nbd 2024-03-29T10:34:08.479 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:56 ago on Fri 29 Mar 2024 10:32:12 AM UTC. 2024-03-29T10:34:08.628 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T10:34:08.629 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:34:08.629 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-03-29T10:34:08.629 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:34:08.629 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T10:34:08.629 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:34:08.629 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:08.629 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T10:34:08.629 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T10:34:08.630 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-29T10:34:08.630 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:08.630 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 171 k 2024-03-29T10:34:08.630 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 494 k 2024-03-29T10:34:08.630 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T10:34:08.862 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 739 kB/s | 171 kB 00:00 2024-03-29T10:34:08.863 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:34:08.863 INFO:teuthology.orchestra.run.smithi086.stdout:Total 734 kB/s | 171 kB 00:00 2024-03-29T10:34:08.863 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T10:34:08.871 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T10:34:08.871 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T10:34:08.928 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T10:34:08.928 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T10:34:09.083 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T10:34:09.186 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:34:09.620 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:34:09.891 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:34:09.891 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:09.891 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T10:34:09.891 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:34:09.891 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:09.891 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T10:34:10.106 DEBUG:teuthology.parallel:result is None 2024-03-29T10:34:10.107 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:34:10.251 DEBUG:teuthology.orchestra.run.smithi084:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:34:10.275 INFO:teuthology.orchestra.run.smithi084.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:34:10.276 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:34:10.276 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:34:10.278 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:34:10.417 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:34:10.438 INFO:teuthology.orchestra.run.smithi086.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:34:10.439 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:34:10.439 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:34:10.443 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T10:34:10.443 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T10:34:10.444 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:34:10.476 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T10:34:10.476 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:34:10.506 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T10:34:10.507 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T10:34:10.507 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:34:10.547 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:34:10.617 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T10:34:10.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:34:10.646 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:34:10.713 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T10:34:10.713 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T10:34:10.713 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:34:10.742 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:34:10.810 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T10:34:10.810 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:34:10.837 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:34:10.904 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T10:34:10.904 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T10:34:10.904 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:34:10.932 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:34:11.001 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T10:34:11.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:34:11.028 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:34:11.095 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T10:34:11.177 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T10:34:11.178 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 777 /var/log/ceph 2024-03-29T10:34:11.180 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 777 /var/log/ceph 2024-03-29T10:34:11.207 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T10:34:11.207 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:34:11.248 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:34:11.275 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T10:34:11.275 DEBUG:teuthology.orchestra.run.smithi084:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:34:11.317 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:34:11.353 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T10:34:11.353 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T10:34:11.354 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622642', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622642', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi084.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMXl6Z9hk+MhmxBMtW9InRHY/lu+i+im1gzZI1WaqF17XERyUg+tOpwSRjG/+63JuJhTlOL2azv4V46SqomePKE=', 'smithi086.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE0cRLpkX/ZRNRgRxmItsxmFa4K2SATDuO+KjNERXWH7L6Wu1GBemAyH1YSSIGIkyoPht418Dx+hVVZEPMw6gOg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T10:34:11.354 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:34:11.384 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:34:11.399 DEBUG:teuthology.orchestra.run.smithi084:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:34:11.440 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:34:11.470 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T10:34:11.470 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:34:11.520 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:34:11.521 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_1 2024-03-29T10:34:11.575 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:34:11.576 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:34:11.576 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-29T10:34:11.576 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:34:11.576 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:34:11.576 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-03-29 10:34:02.054840028 +0000 2024-03-29T10:34:11.576 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-03-29 10:31:24.598134231 +0000 2024-03-29T10:34:11.576 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-03-29 10:31:24.598134231 +0000 2024-03-29T10:34:11.576 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: 2024-03-29 10:31:24.598134231 +0000 2024-03-29T10:34:11.576 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:34:11.642 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-03-29T10:34:11.642 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-03-29T10:34:11.642 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000234024 s, 2.2 MB/s 2024-03-29T10:34:11.643 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:34:11.699 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_2 2024-03-29T10:34:11.754 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:34:11.754 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:34:11.754 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-29T10:34:11.754 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:34:11.754 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:34:11.754 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-03-29 10:34:02.054840028 +0000 2024-03-29T10:34:11.754 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-03-29 10:31:24.942128181 +0000 2024-03-29T10:34:11.754 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-03-29 10:31:24.942128181 +0000 2024-03-29T10:34:11.754 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: 2024-03-29 10:31:24.942128181 +0000 2024-03-29T10:34:11.755 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:34:11.819 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-03-29T10:34:11.819 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-03-29T10:34:11.819 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000214854 s, 2.4 MB/s 2024-03-29T10:34:11.820 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:34:11.878 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_3 2024-03-29T10:34:11.933 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:34:11.933 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:34:11.933 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-29T10:34:11.933 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:34:11.934 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:34:11.934 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-03-29 10:34:02.054840028 +0000 2024-03-29T10:34:11.934 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-03-29 10:31:25.298121923 +0000 2024-03-29T10:34:11.934 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-03-29 10:31:25.298121923 +0000 2024-03-29T10:34:11.934 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: 2024-03-29 10:31:25.298121923 +0000 2024-03-29T10:34:11.934 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:34:11.998 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-03-29T10:34:11.999 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-03-29T10:34:11.999 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000191697 s, 2.7 MB/s 2024-03-29T10:34:12.000 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:34:12.055 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_4 2024-03-29T10:34:12.111 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:34:12.111 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:34:12.111 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-29T10:34:12.111 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:34:12.111 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:34:12.111 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-03-29 10:34:02.055840011 +0000 2024-03-29T10:34:12.111 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-03-29 10:31:25.623116220 +0000 2024-03-29T10:34:12.111 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-03-29 10:31:25.623116220 +0000 2024-03-29T10:34:12.111 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: 2024-03-29 10:31:25.623116220 +0000 2024-03-29T10:34:12.112 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:34:12.176 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-03-29T10:34:12.176 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-03-29T10:34:12.176 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000172181 s, 3.0 MB/s 2024-03-29T10:34:12.177 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:34:12.232 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:34:12.232 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T10:34:12.233 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:34:12.249 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:34:12.250 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_1 2024-03-29T10:34:12.306 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:34:12.306 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:34:12.307 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-29T10:34:12.307 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:34:12.307 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:34:12.307 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-03-29 10:34:06.704729551 +0000 2024-03-29T10:34:12.307 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-03-29 10:31:10.140888342 +0000 2024-03-29T10:34:12.307 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-03-29 10:31:10.140888342 +0000 2024-03-29T10:34:12.307 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-03-29 10:31:10.140888342 +0000 2024-03-29T10:34:12.307 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:34:12.374 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-03-29T10:34:12.374 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-03-29T10:34:12.374 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000249886 s, 2.0 MB/s 2024-03-29T10:34:12.375 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:34:12.433 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_2 2024-03-29T10:34:12.490 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:34:12.490 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:34:12.490 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-29T10:34:12.490 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:34:12.490 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:34:12.490 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-03-29 10:34:06.705729531 +0000 2024-03-29T10:34:12.491 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-03-29 10:31:10.478881641 +0000 2024-03-29T10:34:12.491 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-03-29 10:31:10.478881641 +0000 2024-03-29T10:34:12.491 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-03-29 10:31:10.478881641 +0000 2024-03-29T10:34:12.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:34:12.557 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-03-29T10:34:12.557 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-03-29T10:34:12.557 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.00017717 s, 2.9 MB/s 2024-03-29T10:34:12.558 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:34:12.615 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_3 2024-03-29T10:34:12.670 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:34:12.671 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:34:12.671 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-29T10:34:12.671 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:34:12.671 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:34:12.671 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-03-29 10:34:06.706729511 +0000 2024-03-29T10:34:12.671 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-03-29 10:31:10.818874900 +0000 2024-03-29T10:34:12.671 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-03-29 10:31:10.818874900 +0000 2024-03-29T10:34:12.671 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-03-29 10:31:10.818874900 +0000 2024-03-29T10:34:12.671 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:34:12.737 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-03-29T10:34:12.737 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-03-29T10:34:12.737 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000184 s, 2.8 MB/s 2024-03-29T10:34:12.738 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:34:12.794 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_4 2024-03-29T10:34:12.849 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:34:12.849 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:34:12.850 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T10:34:12.850 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:34:12.850 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:34:12.850 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-03-29 10:34:06.706729511 +0000 2024-03-29T10:34:12.850 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-03-29 10:31:11.174867842 +0000 2024-03-29T10:34:12.850 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-03-29 10:31:11.174867842 +0000 2024-03-29T10:34:12.850 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-03-29 10:31:11.174867842 +0000 2024-03-29T10:34:12.850 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:34:12.915 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-03-29T10:34:12.915 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-03-29T10:34:12.915 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000246199 s, 2.1 MB/s 2024-03-29T10:34:12.916 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:34:12.971 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:34:12.972 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi084.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi086.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:34:12.972 INFO:tasks.ceph:Generating config... 2024-03-29T10:34:12.973 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T10:34:12.973 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T10:34:12.973 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T10:34:12.973 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T10:34:12.973 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T10:34:12.973 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T10:34:12.973 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T10:34:12.973 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T10:34:12.974 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T10:34:12.974 INFO:tasks.ceph:[global] ms type = async 2024-03-29T10:34:12.974 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T10:34:12.974 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T10:34:12.974 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T10:34:12.974 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T10:34:12.974 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T10:34:12.974 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T10:34:12.974 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T10:34:12.974 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T10:34:12.974 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T10:34:12.974 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T10:34:12.974 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T10:34:12.974 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T10:34:12.974 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T10:34:12.975 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-03-29T10:34:12.975 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T10:34:12.975 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T10:34:12.975 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T10:34:12.975 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T10:34:12.975 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T10:34:12.975 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T10:34:12.975 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T10:34:12.975 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T10:34:12.975 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T10:34:12.975 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T10:34:12.975 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T10:34:12.975 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T10:34:12.975 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T10:34:12.975 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T10:34:12.975 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T10:34:12.976 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T10:34:12.976 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T10:34:12.976 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T10:34:12.976 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T10:34:12.976 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T10:34:12.976 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T10:34:12.976 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T10:34:12.976 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T10:34:12.976 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T10:34:12.976 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T10:34:12.976 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T10:34:13.027 INFO:teuthology.orchestra.run.smithi084.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T10:34:13.030 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T10:34:13.075 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:34:13.101 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.84:6789]'), ('mon.c', '[v1:172.21.15.84:6790]'), ('mon.b', '[v1:172.21.15.86:6789]')] 2024-03-29T10:34:13.102 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.84:6789],[v1:172.21.15.84:6790],[v1:172.21.15.86:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T10:34:13.103 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T10:34:13.103 DEBUG:teuthology.orchestra.run.smithi084:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:34:13.156 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.84:6789]' --addv c '[v1:172.21.15.84:6790]' --addv b '[v1:172.21.15.86:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:34:13.240 INFO:teuthology.orchestra.run.smithi084.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:34:13.241 INFO:teuthology.orchestra.run.smithi084.stdout:monmaptool: generated fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:13.241 INFO:teuthology.orchestra.run.smithi084.stdout:setting min_mon_release = quincy 2024-03-29T10:34:13.241 INFO:teuthology.orchestra.run.smithi084.stdout:epoch 0 2024-03-29T10:34:13.241 INFO:teuthology.orchestra.run.smithi084.stdout:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:13.241 INFO:teuthology.orchestra.run.smithi084.stdout:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:13.241 INFO:teuthology.orchestra.run.smithi084.stdout:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:13.241 INFO:teuthology.orchestra.run.smithi084.stdout:min_mon_release 17 (quincy) 2024-03-29T10:34:13.241 INFO:teuthology.orchestra.run.smithi084.stdout:election_strategy: 1 2024-03-29T10:34:13.241 INFO:teuthology.orchestra.run.smithi084.stdout:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:13.241 INFO:teuthology.orchestra.run.smithi084.stdout:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:13.242 INFO:teuthology.orchestra.run.smithi084.stdout:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:13.242 INFO:teuthology.orchestra.run.smithi084.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T10:34:13.243 DEBUG:teuthology.orchestra.run.smithi084:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:34:13.257 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 88c2d849-612c-4e38-a628-0989e024a981... 2024-03-29T10:34:13.258 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:34:13.299 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:34:13.348 INFO:teuthology.orchestra.run.smithi084.stdout:[global] 2024-03-29T10:34:13.348 INFO:teuthology.orchestra.run.smithi084.stdout: chdir = "" 2024-03-29T10:34:13.348 INFO:teuthology.orchestra.run.smithi084.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:34:13.348 INFO:teuthology.orchestra.run.smithi084.stdout: auth supported = cephx 2024-03-29T10:34:13.348 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:13.348 INFO:teuthology.orchestra.run.smithi084.stdout: filestore xattr use omap = true 2024-03-29T10:34:13.348 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:13.348 INFO:teuthology.orchestra.run.smithi084.stdout: mon clock drift allowed = 1.000 2024-03-29T10:34:13.348 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:13.348 INFO:teuthology.orchestra.run.smithi084.stdout: osd crush chooseleaf type = 0 2024-03-29T10:34:13.348 INFO:teuthology.orchestra.run.smithi084.stdout: auth debug = true 2024-03-29T10:34:13.349 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:13.349 INFO:teuthology.orchestra.run.smithi084.stdout: ms die on old message = true 2024-03-29T10:34:13.349 INFO:teuthology.orchestra.run.smithi084.stdout: ms die on bug = true 2024-03-29T10:34:13.349 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:13.349 INFO:teuthology.orchestra.run.smithi084.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:34:13.349 INFO:teuthology.orchestra.run.smithi084.stdout: mon pg warn max object skew = 0 2024-03-29T10:34:13.349 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:13.349 INFO:teuthology.orchestra.run.smithi084.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:34:13.349 INFO:teuthology.orchestra.run.smithi084.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:34:13.349 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:13.349 INFO:teuthology.orchestra.run.smithi084.stdout: osd pool default size = 2 2024-03-29T10:34:13.349 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:13.349 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd allow primary affinity = true 2024-03-29T10:34:13.349 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd allow pg remap = true 2024-03-29T10:34:13.349 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:34:13.350 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:34:13.350 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on no sortbitwise = false 2024-03-29T10:34:13.350 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:34:13.350 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on too few osds = false 2024-03-29T10:34:13.350 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:34:13.350 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:34:13.350 INFO:teuthology.orchestra.run.smithi084.stdout: mon_allow_pool_size_one = true 2024-03-29T10:34:13.350 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:13.350 INFO:teuthology.orchestra.run.smithi084.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:34:13.350 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:13.350 INFO:teuthology.orchestra.run.smithi084.stdout: osd default data pool replay window = 5 2024-03-29T10:34:13.350 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:13.350 INFO:teuthology.orchestra.run.smithi084.stdout: mon allow pool delete = true 2024-03-29T10:34:13.350 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:13.351 INFO:teuthology.orchestra.run.smithi084.stdout: mon cluster log file level = debug 2024-03-29T10:34:13.351 INFO:teuthology.orchestra.run.smithi084.stdout: debug asserts on shutdown = true 2024-03-29T10:34:13.351 INFO:teuthology.orchestra.run.smithi084.stdout: mon health detail to clog = false 2024-03-29T10:34:13.351 INFO:teuthology.orchestra.run.smithi084.stdout: mon host = "[v1:172.21.15.84:6789],[v1:172.21.15.84:6790],[v1:172.21.15.86:6789]" 2024-03-29T10:34:13.351 INFO:teuthology.orchestra.run.smithi084.stdout: lockdep = True 2024-03-29T10:34:13.351 INFO:teuthology.orchestra.run.smithi084.stdout: mon client directed command retry = 5 2024-03-29T10:34:13.351 INFO:teuthology.orchestra.run.smithi084.stdout: mon election default strategy = 1 2024-03-29T10:34:13.351 INFO:teuthology.orchestra.run.smithi084.stdout: ms bind msgr2 = False 2024-03-29T10:34:13.351 INFO:teuthology.orchestra.run.smithi084.stdout: ms inject socket failures = 5000 2024-03-29T10:34:13.351 INFO:teuthology.orchestra.run.smithi084.stdout: ms type = async 2024-03-29T10:34:13.351 INFO:teuthology.orchestra.run.smithi084.stdout: fsid = 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:13.351 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:13.351 INFO:teuthology.orchestra.run.smithi084.stdout:[osd] 2024-03-29T10:34:13.351 INFO:teuthology.orchestra.run.smithi084.stdout: osd journal size = 100 2024-03-29T10:34:13.352 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:13.352 INFO:teuthology.orchestra.run.smithi084.stdout: osd scrub load threshold = 5.0 2024-03-29T10:34:13.352 INFO:teuthology.orchestra.run.smithi084.stdout: osd scrub max interval = 600 2024-03-29T10:34:13.352 INFO:teuthology.orchestra.run.smithi084.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:34:13.352 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:13.352 INFO:teuthology.orchestra.run.smithi084.stdout: osd recover clone overlap = true 2024-03-29T10:34:13.352 INFO:teuthology.orchestra.run.smithi084.stdout: osd recovery max chunk = 1048576 2024-03-29T10:34:13.352 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:13.352 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug shutdown = true 2024-03-29T10:34:13.352 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug op order = true 2024-03-29T10:34:13.352 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug verify stray on activate = true 2024-03-29T10:34:13.352 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug trim objects = true 2024-03-29T10:34:13.352 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:13.352 INFO:teuthology.orchestra.run.smithi084.stdout: osd open classes on start = true 2024-03-29T10:34:13.353 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug pg log writeout = true 2024-03-29T10:34:13.353 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:13.353 INFO:teuthology.orchestra.run.smithi084.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:34:13.353 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:13.353 INFO:teuthology.orchestra.run.smithi084.stdout: osd map max advance = 10 2024-03-29T10:34:13.353 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:13.353 INFO:teuthology.orchestra.run.smithi084.stdout: journal zero on create = true 2024-03-29T10:34:13.353 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:13.353 INFO:teuthology.orchestra.run.smithi084.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:34:13.353 INFO:teuthology.orchestra.run.smithi084.stdout: filestore apply finisher threads = 3 2024-03-29T10:34:13.353 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:13.353 INFO:teuthology.orchestra.run.smithi084.stdout: bdev debug aio = true 2024-03-29T10:34:13.353 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug misdirected ops = true 2024-03-29T10:34:13.353 INFO:teuthology.orchestra.run.smithi084.stdout: bluestore block size = 96636764160 2024-03-29T10:34:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: bluestore compression algorithm = zstd 2024-03-29T10:34:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: bluestore compression mode = aggressive 2024-03-29T10:34:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: bluestore fsck on mount = True 2024-03-29T10:34:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: bluestore zero block detection = True 2024-03-29T10:34:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: debug bluefs = 20 2024-03-29T10:34:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: debug bluestore = 20 2024-03-29T10:34:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: debug monc = 20 2024-03-29T10:34:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: debug ms = 1 2024-03-29T10:34:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: debug osd = 20 2024-03-29T10:34:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: debug rocksdb = 10 2024-03-29T10:34:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:34:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd full ratio = 0.9 2024-03-29T10:34:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:34:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: osd class default list = * 2024-03-29T10:34:13.354 INFO:teuthology.orchestra.run.smithi084.stdout: osd class load list = * 2024-03-29T10:34:13.355 INFO:teuthology.orchestra.run.smithi084.stdout: osd client watch timeout = 120 2024-03-29T10:34:13.355 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug verify cached snaps = True 2024-03-29T10:34:13.355 INFO:teuthology.orchestra.run.smithi084.stdout: osd debug verify missing on start = True 2024-03-29T10:34:13.355 INFO:teuthology.orchestra.run.smithi084.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:34:13.355 INFO:teuthology.orchestra.run.smithi084.stdout: osd mclock override recovery settings = True 2024-03-29T10:34:13.355 INFO:teuthology.orchestra.run.smithi084.stdout: osd objectstore = bluestore 2024-03-29T10:34:13.355 INFO:teuthology.orchestra.run.smithi084.stdout: osd op queue = debug_random 2024-03-29T10:34:13.355 INFO:teuthology.orchestra.run.smithi084.stdout: osd op queue cut off = debug_random 2024-03-29T10:34:13.355 INFO:teuthology.orchestra.run.smithi084.stdout: osd shutdown pgref assert = True 2024-03-29T10:34:13.355 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:13.355 INFO:teuthology.orchestra.run.smithi084.stdout:[mgr] 2024-03-29T10:34:13.355 INFO:teuthology.orchestra.run.smithi084.stdout: debug ms = 1 2024-03-29T10:34:13.355 INFO:teuthology.orchestra.run.smithi084.stdout: debug mgr = 20 2024-03-29T10:34:13.355 INFO:teuthology.orchestra.run.smithi084.stdout: debug mon = 20 2024-03-29T10:34:13.355 INFO:teuthology.orchestra.run.smithi084.stdout: debug auth = 20 2024-03-29T10:34:13.356 INFO:teuthology.orchestra.run.smithi084.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:34:13.356 INFO:teuthology.orchestra.run.smithi084.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:34:13.356 INFO:teuthology.orchestra.run.smithi084.stdout: mgr/telemetry/nag = false 2024-03-29T10:34:13.356 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:13.356 INFO:teuthology.orchestra.run.smithi084.stdout:[mon] 2024-03-29T10:34:13.356 INFO:teuthology.orchestra.run.smithi084.stdout: debug ms = 1 2024-03-29T10:34:13.356 INFO:teuthology.orchestra.run.smithi084.stdout: debug mon = 20 2024-03-29T10:34:13.356 INFO:teuthology.orchestra.run.smithi084.stdout: debug paxos = 20 2024-03-29T10:34:13.356 INFO:teuthology.orchestra.run.smithi084.stdout: debug auth = 20 2024-03-29T10:34:13.356 INFO:teuthology.orchestra.run.smithi084.stdout: mon data avail warn = 5 2024-03-29T10:34:13.356 INFO:teuthology.orchestra.run.smithi084.stdout: mon mgr mkfs grace = 240 2024-03-29T10:34:13.356 INFO:teuthology.orchestra.run.smithi084.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:34:13.356 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd reporter subtree level = osd 2024-03-29T10:34:13.356 INFO:teuthology.orchestra.run.smithi084.stdout: mon osd prime pg temp = true 2024-03-29T10:34:13.356 INFO:teuthology.orchestra.run.smithi084.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:34:13.357 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:13.357 INFO:teuthology.orchestra.run.smithi084.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:34:13.357 INFO:teuthology.orchestra.run.smithi084.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:34:13.357 INFO:teuthology.orchestra.run.smithi084.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:34:13.357 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:13.357 INFO:teuthology.orchestra.run.smithi084.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:34:13.357 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:34:13.357 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:34:13.357 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:13.357 INFO:teuthology.orchestra.run.smithi084.stdout: # 1m isn't quite enough 2024-03-29T10:34:13.357 INFO:teuthology.orchestra.run.smithi084.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:34:13.357 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:13.357 INFO:teuthology.orchestra.run.smithi084.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:34:13.357 INFO:teuthology.orchestra.run.smithi084.stdout: mon min osdmap epochs = 50 2024-03-29T10:34:13.357 INFO:teuthology.orchestra.run.smithi084.stdout: mon osdmap full prune interval = 2 2024-03-29T10:34:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: mon osdmap full prune min = 15 2024-03-29T10:34:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:34:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: mon scrub interval = 300 2024-03-29T10:34:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: mon warn on pool no app = False 2024-03-29T10:34:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: paxos service trim min = 10 2024-03-29T10:34:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:13.358 INFO:teuthology.orchestra.run.smithi084.stdout:[client] 2024-03-29T10:34:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: rgw cache enabled = true 2024-03-29T10:34:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: rgw enable ops log = true 2024-03-29T10:34:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: rgw enable usage log = true 2024-03-29T10:34:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:34:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:34:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: debug monc = 20 2024-03-29T10:34:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: debug ms = 1 2024-03-29T10:34:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: debug objecter = 20 2024-03-29T10:34:13.358 INFO:teuthology.orchestra.run.smithi084.stdout: debug rados = 20 2024-03-29T10:34:13.359 INFO:teuthology.orchestra.run.smithi084.stdout:[mon.a] 2024-03-29T10:34:13.359 INFO:teuthology.orchestra.run.smithi084.stdout:[mon.c] 2024-03-29T10:34:13.359 INFO:teuthology.orchestra.run.smithi084.stdout:[mon.b] 2024-03-29T10:34:13.359 INFO:teuthology.orchestra.run.smithi086.stdout:[global] 2024-03-29T10:34:13.359 INFO:teuthology.orchestra.run.smithi086.stdout: chdir = "" 2024-03-29T10:34:13.359 INFO:teuthology.orchestra.run.smithi086.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:34:13.359 INFO:teuthology.orchestra.run.smithi086.stdout: auth supported = cephx 2024-03-29T10:34:13.359 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:13.359 INFO:teuthology.orchestra.run.smithi086.stdout: filestore xattr use omap = true 2024-03-29T10:34:13.360 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:13.360 INFO:teuthology.orchestra.run.smithi086.stdout: mon clock drift allowed = 1.000 2024-03-29T10:34:13.360 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:13.360 INFO:teuthology.orchestra.run.smithi086.stdout: osd crush chooseleaf type = 0 2024-03-29T10:34:13.360 INFO:teuthology.orchestra.run.smithi086.stdout: auth debug = true 2024-03-29T10:34:13.360 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:13.360 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on old message = true 2024-03-29T10:34:13.360 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on bug = true 2024-03-29T10:34:13.360 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:13.360 INFO:teuthology.orchestra.run.smithi086.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:34:13.360 INFO:teuthology.orchestra.run.smithi086.stdout: mon pg warn max object skew = 0 2024-03-29T10:34:13.360 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:13.360 INFO:teuthology.orchestra.run.smithi086.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:34:13.360 INFO:teuthology.orchestra.run.smithi086.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:34:13.361 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:13.361 INFO:teuthology.orchestra.run.smithi086.stdout: osd pool default size = 2 2024-03-29T10:34:13.361 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:13.361 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd allow primary affinity = true 2024-03-29T10:34:13.361 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd allow pg remap = true 2024-03-29T10:34:13.361 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:34:13.361 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:34:13.361 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on no sortbitwise = false 2024-03-29T10:34:13.361 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:34:13.361 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on too few osds = false 2024-03-29T10:34:13.361 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:34:13.361 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:34:13.361 INFO:teuthology.orchestra.run.smithi086.stdout: mon_allow_pool_size_one = true 2024-03-29T10:34:13.361 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:13.362 INFO:teuthology.orchestra.run.smithi086.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:34:13.362 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:13.362 INFO:teuthology.orchestra.run.smithi086.stdout: osd default data pool replay window = 5 2024-03-29T10:34:13.362 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:13.362 INFO:teuthology.orchestra.run.smithi086.stdout: mon allow pool delete = true 2024-03-29T10:34:13.362 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:13.362 INFO:teuthology.orchestra.run.smithi086.stdout: mon cluster log file level = debug 2024-03-29T10:34:13.362 INFO:teuthology.orchestra.run.smithi086.stdout: debug asserts on shutdown = true 2024-03-29T10:34:13.362 INFO:teuthology.orchestra.run.smithi086.stdout: mon health detail to clog = false 2024-03-29T10:34:13.362 INFO:teuthology.orchestra.run.smithi086.stdout: mon host = "[v1:172.21.15.84:6789],[v1:172.21.15.84:6790],[v1:172.21.15.86:6789]" 2024-03-29T10:34:13.362 INFO:teuthology.orchestra.run.smithi086.stdout: lockdep = True 2024-03-29T10:34:13.362 INFO:teuthology.orchestra.run.smithi086.stdout: mon client directed command retry = 5 2024-03-29T10:34:13.362 INFO:teuthology.orchestra.run.smithi086.stdout: mon election default strategy = 1 2024-03-29T10:34:13.363 INFO:teuthology.orchestra.run.smithi086.stdout: ms bind msgr2 = False 2024-03-29T10:34:13.363 INFO:teuthology.orchestra.run.smithi086.stdout: ms inject socket failures = 5000 2024-03-29T10:34:13.363 INFO:teuthology.orchestra.run.smithi086.stdout: ms type = async 2024-03-29T10:34:13.363 INFO:teuthology.orchestra.run.smithi086.stdout: fsid = 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:13.363 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:13.363 INFO:teuthology.orchestra.run.smithi086.stdout:[osd] 2024-03-29T10:34:13.363 INFO:teuthology.orchestra.run.smithi086.stdout: osd journal size = 100 2024-03-29T10:34:13.363 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:13.363 INFO:teuthology.orchestra.run.smithi086.stdout: osd scrub load threshold = 5.0 2024-03-29T10:34:13.363 INFO:teuthology.orchestra.run.smithi086.stdout: osd scrub max interval = 600 2024-03-29T10:34:13.363 INFO:teuthology.orchestra.run.smithi086.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:34:13.363 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:13.363 INFO:teuthology.orchestra.run.smithi086.stdout: osd recover clone overlap = true 2024-03-29T10:34:13.363 INFO:teuthology.orchestra.run.smithi086.stdout: osd recovery max chunk = 1048576 2024-03-29T10:34:13.363 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:13.364 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug shutdown = true 2024-03-29T10:34:13.364 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug op order = true 2024-03-29T10:34:13.364 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug verify stray on activate = true 2024-03-29T10:34:13.364 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug trim objects = true 2024-03-29T10:34:13.364 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:13.364 INFO:teuthology.orchestra.run.smithi086.stdout: osd open classes on start = true 2024-03-29T10:34:13.364 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug pg log writeout = true 2024-03-29T10:34:13.364 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:13.364 INFO:teuthology.orchestra.run.smithi086.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:34:13.364 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:13.364 INFO:teuthology.orchestra.run.smithi086.stdout: osd map max advance = 10 2024-03-29T10:34:13.364 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:13.364 INFO:teuthology.orchestra.run.smithi086.stdout: journal zero on create = true 2024-03-29T10:34:13.364 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:13.365 INFO:teuthology.orchestra.run.smithi086.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:34:13.365 INFO:teuthology.orchestra.run.smithi086.stdout: filestore apply finisher threads = 3 2024-03-29T10:34:13.365 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:13.365 INFO:teuthology.orchestra.run.smithi086.stdout: bdev debug aio = true 2024-03-29T10:34:13.365 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug misdirected ops = true 2024-03-29T10:34:13.365 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore block size = 96636764160 2024-03-29T10:34:13.365 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore compression algorithm = zstd 2024-03-29T10:34:13.365 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore compression mode = aggressive 2024-03-29T10:34:13.365 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore fsck on mount = True 2024-03-29T10:34:13.365 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore zero block detection = True 2024-03-29T10:34:13.365 INFO:teuthology.orchestra.run.smithi086.stdout: debug bluefs = 20 2024-03-29T10:34:13.365 INFO:teuthology.orchestra.run.smithi086.stdout: debug bluestore = 20 2024-03-29T10:34:13.365 INFO:teuthology.orchestra.run.smithi086.stdout: debug monc = 20 2024-03-29T10:34:13.365 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2024-03-29T10:34:13.365 INFO:teuthology.orchestra.run.smithi086.stdout: debug osd = 20 2024-03-29T10:34:13.365 INFO:teuthology.orchestra.run.smithi086.stdout: debug rocksdb = 10 2024-03-29T10:34:13.366 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:34:13.366 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd full ratio = 0.9 2024-03-29T10:34:13.366 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:34:13.366 INFO:teuthology.orchestra.run.smithi086.stdout: osd class default list = * 2024-03-29T10:34:13.366 INFO:teuthology.orchestra.run.smithi086.stdout: osd class load list = * 2024-03-29T10:34:13.366 INFO:teuthology.orchestra.run.smithi086.stdout: osd client watch timeout = 120 2024-03-29T10:34:13.366 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug verify cached snaps = True 2024-03-29T10:34:13.366 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug verify missing on start = True 2024-03-29T10:34:13.366 INFO:teuthology.orchestra.run.smithi086.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:34:13.366 INFO:teuthology.orchestra.run.smithi086.stdout: osd mclock override recovery settings = True 2024-03-29T10:34:13.366 INFO:teuthology.orchestra.run.smithi086.stdout: osd objectstore = bluestore 2024-03-29T10:34:13.366 INFO:teuthology.orchestra.run.smithi086.stdout: osd op queue = debug_random 2024-03-29T10:34:13.366 INFO:teuthology.orchestra.run.smithi086.stdout: osd op queue cut off = debug_random 2024-03-29T10:34:13.366 INFO:teuthology.orchestra.run.smithi086.stdout: osd shutdown pgref assert = True 2024-03-29T10:34:13.367 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:13.367 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr] 2024-03-29T10:34:13.367 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2024-03-29T10:34:13.367 INFO:teuthology.orchestra.run.smithi086.stdout: debug mgr = 20 2024-03-29T10:34:13.367 INFO:teuthology.orchestra.run.smithi086.stdout: debug mon = 20 2024-03-29T10:34:13.367 INFO:teuthology.orchestra.run.smithi086.stdout: debug auth = 20 2024-03-29T10:34:13.367 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:34:13.367 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:34:13.367 INFO:teuthology.orchestra.run.smithi086.stdout: mgr/telemetry/nag = false 2024-03-29T10:34:13.367 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:13.367 INFO:teuthology.orchestra.run.smithi086.stdout:[mon] 2024-03-29T10:34:13.367 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2024-03-29T10:34:13.367 INFO:teuthology.orchestra.run.smithi086.stdout: debug mon = 20 2024-03-29T10:34:13.367 INFO:teuthology.orchestra.run.smithi086.stdout: debug paxos = 20 2024-03-29T10:34:13.367 INFO:teuthology.orchestra.run.smithi086.stdout: debug auth = 20 2024-03-29T10:34:13.367 INFO:teuthology.orchestra.run.smithi086.stdout: mon data avail warn = 5 2024-03-29T10:34:13.368 INFO:teuthology.orchestra.run.smithi086.stdout: mon mgr mkfs grace = 240 2024-03-29T10:34:13.368 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:34:13.368 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd reporter subtree level = osd 2024-03-29T10:34:13.368 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd prime pg temp = true 2024-03-29T10:34:13.368 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:34:13.368 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:13.368 INFO:teuthology.orchestra.run.smithi086.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:34:13.368 INFO:teuthology.orchestra.run.smithi086.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:34:13.368 INFO:teuthology.orchestra.run.smithi086.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:34:13.368 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:13.368 INFO:teuthology.orchestra.run.smithi086.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:34:13.368 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:34:13.368 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:34:13.368 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:13.368 INFO:teuthology.orchestra.run.smithi086.stdout: # 1m isn't quite enough 2024-03-29T10:34:13.369 INFO:teuthology.orchestra.run.smithi086.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:34:13.369 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:13.369 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:34:13.369 INFO:teuthology.orchestra.run.smithi086.stdout: mon min osdmap epochs = 50 2024-03-29T10:34:13.369 INFO:teuthology.orchestra.run.smithi086.stdout: mon osdmap full prune interval = 2 2024-03-29T10:34:13.369 INFO:teuthology.orchestra.run.smithi086.stdout: mon osdmap full prune min = 15 2024-03-29T10:34:13.369 INFO:teuthology.orchestra.run.smithi086.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:34:13.369 INFO:teuthology.orchestra.run.smithi086.stdout: mon scrub interval = 300 2024-03-29T10:34:13.369 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on pool no app = False 2024-03-29T10:34:13.369 INFO:teuthology.orchestra.run.smithi086.stdout: paxos service trim min = 10 2024-03-29T10:34:13.369 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T10:34:13.369 INFO:teuthology.orchestra.run.smithi086.stdout:[client] 2024-03-29T10:34:13.369 INFO:teuthology.orchestra.run.smithi086.stdout: rgw cache enabled = true 2024-03-29T10:34:13.369 INFO:teuthology.orchestra.run.smithi086.stdout: rgw enable ops log = true 2024-03-29T10:34:13.369 INFO:teuthology.orchestra.run.smithi086.stdout: rgw enable usage log = true 2024-03-29T10:34:13.369 INFO:teuthology.orchestra.run.smithi086.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:34:13.370 INFO:teuthology.orchestra.run.smithi086.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:34:13.370 INFO:teuthology.orchestra.run.smithi086.stdout: debug monc = 20 2024-03-29T10:34:13.370 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2024-03-29T10:34:13.370 INFO:teuthology.orchestra.run.smithi086.stdout: debug objecter = 20 2024-03-29T10:34:13.370 INFO:teuthology.orchestra.run.smithi086.stdout: debug rados = 20 2024-03-29T10:34:13.370 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.a] 2024-03-29T10:34:13.370 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.c] 2024-03-29T10:34:13.370 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.b] 2024-03-29T10:34:13.371 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T10:34:13.372 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T10:34:13.466 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T10:34:13.466 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T10:34:13.466 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T10:34:13.481 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T10:34:13.481 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T10:34:13.536 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi084.front.sepia.ceph.com 2024-03-29T10:34:13.536 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T10:34:13.536 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:34:13.536 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:34:13.615 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T10:34:13.615 DEBUG:teuthology.orchestra.run.smithi084:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:34:13.669 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi086.front.sepia.ceph.com 2024-03-29T10:34:13.670 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T10:34:13.670 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:34:13.670 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:34:13.709 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T10:34:13.710 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:34:13.764 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T10:34:13.764 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T10:34:13.765 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:34:13.819 INFO:teuthology.orchestra.run.smithi084.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:34:13.821 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:34:13.889 INFO:teuthology.orchestra.run.smithi086.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:34:13.892 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T10:34:13.892 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T10:34:13.892 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T10:34:13.935 INFO:teuthology.orchestra.run.smithi084.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T10:34:13.949 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T10:34:13.991 INFO:teuthology.orchestra.run.smithi086.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T10:34:14.006 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T10:34:14.007 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi084.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi086.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:34:14.007 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T10:34:14.033 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:34:14.033 INFO:tasks.ceph:role: osd.0 2024-03-29T10:34:14.033 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-29T10:34:14.033 DEBUG:teuthology.orchestra.run.smithi084:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:34:14.107 INFO:teuthology.orchestra.run.smithi084.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:34:14.107 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:34:14.107 INFO:teuthology.orchestra.run.smithi084.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:34:14.107 INFO:teuthology.orchestra.run.smithi084.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:34:14.107 INFO:teuthology.orchestra.run.smithi084.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:34:14.107 INFO:teuthology.orchestra.run.smithi084.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:34:14.107 INFO:teuthology.orchestra.run.smithi084.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:34:14.108 INFO:teuthology.orchestra.run.smithi084.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:34:14.108 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:34:14.108 INFO:teuthology.orchestra.run.smithi084.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:34:14.577 INFO:teuthology.orchestra.run.smithi084.stdout:Discarding blocks...Done. 2024-03-29T10:34:14.579 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi084.front.sepia.ceph.com -o noatime 2024-03-29T10:34:14.579 DEBUG:teuthology.orchestra.run.smithi084:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T10:34:14.694 DEBUG:teuthology.orchestra.run.smithi084:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T10:34:14.729 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T10:34:14.796 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:34:14.797 INFO:tasks.ceph:role: osd.1 2024-03-29T10:34:14.797 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-29T10:34:14.797 DEBUG:teuthology.orchestra.run.smithi084:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:34:14.873 INFO:teuthology.orchestra.run.smithi084.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:34:14.873 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:34:14.873 INFO:teuthology.orchestra.run.smithi084.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:34:14.873 INFO:teuthology.orchestra.run.smithi084.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:34:14.873 INFO:teuthology.orchestra.run.smithi084.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:34:14.873 INFO:teuthology.orchestra.run.smithi084.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:34:14.873 INFO:teuthology.orchestra.run.smithi084.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:34:14.873 INFO:teuthology.orchestra.run.smithi084.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:34:14.873 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:34:14.874 INFO:teuthology.orchestra.run.smithi084.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:34:15.332 INFO:teuthology.orchestra.run.smithi084.stdout:Discarding blocks...Done. 2024-03-29T10:34:15.334 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi084.front.sepia.ceph.com -o noatime 2024-03-29T10:34:15.334 DEBUG:teuthology.orchestra.run.smithi084:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T10:34:15.411 DEBUG:teuthology.orchestra.run.smithi084:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T10:34:15.482 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T10:34:15.550 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:34:15.550 INFO:tasks.ceph:role: osd.2 2024-03-29T10:34:15.550 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-29T10:34:15.550 DEBUG:teuthology.orchestra.run.smithi084:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:34:15.624 INFO:teuthology.orchestra.run.smithi084.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:34:15.624 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:34:15.624 INFO:teuthology.orchestra.run.smithi084.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:34:15.624 INFO:teuthology.orchestra.run.smithi084.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:34:15.625 INFO:teuthology.orchestra.run.smithi084.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:34:15.625 INFO:teuthology.orchestra.run.smithi084.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:34:15.625 INFO:teuthology.orchestra.run.smithi084.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:34:15.625 INFO:teuthology.orchestra.run.smithi084.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:34:15.625 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:34:15.625 INFO:teuthology.orchestra.run.smithi084.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:34:16.114 INFO:teuthology.orchestra.run.smithi084.stdout:Discarding blocks...Done. 2024-03-29T10:34:16.116 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi084.front.sepia.ceph.com -o noatime 2024-03-29T10:34:16.116 DEBUG:teuthology.orchestra.run.smithi084:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T10:34:16.212 DEBUG:teuthology.orchestra.run.smithi084:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T10:34:16.243 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T10:34:16.310 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:34:16.310 INFO:tasks.ceph:role: osd.3 2024-03-29T10:34:16.311 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-29T10:34:16.311 DEBUG:teuthology.orchestra.run.smithi084:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:34:16.385 INFO:teuthology.orchestra.run.smithi084.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:34:16.386 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:34:16.386 INFO:teuthology.orchestra.run.smithi084.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:34:16.386 INFO:teuthology.orchestra.run.smithi084.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:34:16.386 INFO:teuthology.orchestra.run.smithi084.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:34:16.386 INFO:teuthology.orchestra.run.smithi084.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:34:16.386 INFO:teuthology.orchestra.run.smithi084.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:34:16.386 INFO:teuthology.orchestra.run.smithi084.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:34:16.386 INFO:teuthology.orchestra.run.smithi084.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:34:16.386 INFO:teuthology.orchestra.run.smithi084.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:34:16.896 INFO:teuthology.orchestra.run.smithi084.stdout:Discarding blocks...Done. 2024-03-29T10:34:16.898 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi084.front.sepia.ceph.com -o noatime 2024-03-29T10:34:16.898 DEBUG:teuthology.orchestra.run.smithi084:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T10:34:17.010 DEBUG:teuthology.orchestra.run.smithi084:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T10:34:17.044 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:34:17.147 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:17.161+0000 7f6220048640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T10:34:17.149 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:17.164+0000 7f6220048640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T10:34:17.149 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:17.164+0000 7f6220048640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:34:17.149 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:17.164+0000 7f6220048640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:34:17.150 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:17.164+0000 7f6220048640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:34:17.150 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:17.165+0000 7f6220048640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T10:34:20.005 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T10:34:20.036 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:34:20.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:20.142+0000 7fea37391640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T10:34:20.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:20.142+0000 7fea37391640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T10:34:20.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:20.142+0000 7fea37391640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:34:20.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:20.142+0000 7fea37391640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:34:20.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:20.142+0000 7fea37391640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:34:20.128 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:20.143+0000 7fea37391640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T10:34:22.970 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T10:34:23.001 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:34:23.092 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:23.106+0000 7f5a1d993640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T10:34:23.092 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:23.106+0000 7f5a1d993640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T10:34:23.092 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:23.106+0000 7f5a1d993640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:34:23.092 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:23.106+0000 7f5a1d993640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:34:23.092 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:23.106+0000 7f5a1d993640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:34:23.092 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:23.107+0000 7f5a1d993640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T10:34:25.939 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T10:34:25.967 DEBUG:teuthology.orchestra.run.smithi084:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:34:26.054 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:26.068+0000 7f8098037640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T10:34:26.054 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:26.068+0000 7f8098037640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T10:34:26.054 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:26.068+0000 7f8098037640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:34:26.054 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:26.068+0000 7f8098037640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:34:26.054 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:26.069+0000 7f8098037640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:34:26.055 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:26.069+0000 7f8098037640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T10:34:28.896 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T10:34:28.927 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T10:34:28.959 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:34:28.959 INFO:tasks.ceph:role: osd.4 2024-03-29T10:34:28.959 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi086.front.sepia.ceph.com 2024-03-29T10:34:28.959 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:34:29.036 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:34:29.036 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:34:29.036 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:34:29.037 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:34:29.037 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:34:29.037 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:34:29.037 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:34:29.037 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:34:29.037 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:34:29.037 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:34:29.552 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2024-03-29T10:34:29.554 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2024-03-29T10:34:29.554 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T10:34:29.684 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T10:34:29.716 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T10:34:29.784 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:34:29.784 INFO:tasks.ceph:role: osd.5 2024-03-29T10:34:29.784 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi086.front.sepia.ceph.com 2024-03-29T10:34:29.784 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:34:29.859 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:34:29.859 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:34:29.859 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:34:29.860 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:34:29.860 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:34:29.860 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:34:29.860 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:34:29.860 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:34:29.860 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:34:29.860 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:34:30.315 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2024-03-29T10:34:30.317 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2024-03-29T10:34:30.317 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T10:34:30.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T10:34:30.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T10:34:30.536 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:34:30.536 INFO:tasks.ceph:role: osd.6 2024-03-29T10:34:30.536 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi086.front.sepia.ceph.com 2024-03-29T10:34:30.536 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:34:30.610 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:34:30.610 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:34:30.610 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:34:30.610 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:34:30.611 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:34:30.611 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:34:30.611 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:34:30.611 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:34:30.611 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:34:30.611 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:34:31.112 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2024-03-29T10:34:31.114 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2024-03-29T10:34:31.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T10:34:31.272 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T10:34:31.305 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T10:34:31.372 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:34:31.372 INFO:tasks.ceph:role: osd.7 2024-03-29T10:34:31.372 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi086.front.sepia.ceph.com 2024-03-29T10:34:31.372 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:34:31.446 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:34:31.446 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:34:31.446 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:34:31.446 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:34:31.447 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:34:31.447 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:34:31.447 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:34:31.447 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:34:31.447 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:34:31.447 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:34:31.934 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2024-03-29T10:34:31.936 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2024-03-29T10:34:31.936 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T10:34:32.054 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T10:34:32.086 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:34:32.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:32.203+0000 7f2cd1ace640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T10:34:32.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:32.203+0000 7f2cd1ace640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T10:34:32.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:32.203+0000 7f2cd1ace640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:34:32.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:32.203+0000 7f2cd1ace640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:34:32.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:32.203+0000 7f2cd1ace640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:34:32.189 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:32.204+0000 7f2cd1ace640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T10:34:35.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T10:34:35.076 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:34:35.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:35.182+0000 7fbadd6c6640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T10:34:35.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:35.182+0000 7fbadd6c6640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T10:34:35.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:35.182+0000 7fbadd6c6640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:34:35.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:35.182+0000 7fbadd6c6640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:34:35.168 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:35.182+0000 7fbadd6c6640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:34:35.169 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:35.183+0000 7fbadd6c6640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T10:34:38.019 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T10:34:38.050 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:34:38.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:38.155+0000 7f2f0ea17640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T10:34:38.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:38.155+0000 7f2f0ea17640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T10:34:38.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:38.155+0000 7f2f0ea17640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:34:38.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:38.155+0000 7f2f0ea17640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:34:38.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:38.155+0000 7f2f0ea17640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:34:38.141 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:38.156+0000 7f2f0ea17640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T10:34:41.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T10:34:41.034 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:34:41.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:41.138+0000 7fa67a010640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T10:34:41.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:41.138+0000 7fa67a010640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T10:34:41.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:41.138+0000 7fa67a010640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:34:41.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:41.138+0000 7fa67a010640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:34:41.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:41.138+0000 7fa67a010640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:34:41.124 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:41.139+0000 7fa67a010640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T10:34:43.979 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T10:34:44.010 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T10:34:44.010 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T10:34:44.010 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T10:34:44.043 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T10:34:44.043 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T10:34:44.113 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T10:34:44.113 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T10:34:44.182 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T10:34:44.182 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T10:34:44.250 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T10:34:44.250 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T10:34:44.317 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T10:34:44.318 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T10:34:44.349 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T10:34:44.349 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T10:34:44.419 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T10:34:44.419 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T10:34:44.488 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T10:34:44.488 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T10:34:44.558 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T10:34:44.558 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T10:34:44.628 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T10:34:44.628 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T10:34:44.644 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T10:34:44.644 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T10:34:44.685 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T10:34:44.685 DEBUG:teuthology.orchestra.run.smithi084:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:34:44.687 DEBUG:teuthology.orchestra.run.smithi086:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:34:44.728 INFO:teuthology.orchestra.run.smithi084.stdout:[mgr.y] 2024-03-29T10:34:44.728 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQAlmQZmzFbEMRAAK99pTr/W15nhiuzAPEPFEw== 2024-03-29T10:34:44.728 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.0] 2024-03-29T10:34:44.728 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQApmQZmVeGxCRAA9Rb4Y73bYUNdg0Cv3fWBjw== 2024-03-29T10:34:44.729 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.1] 2024-03-29T10:34:44.729 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQAsmQZmIcuKCBAAykp98AqF/jxzl6Rs5UYO9g== 2024-03-29T10:34:44.729 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.2] 2024-03-29T10:34:44.729 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQAvmQZmW8ppBhAA+XEceGs4wGhoRcaySx52eA== 2024-03-29T10:34:44.729 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.3] 2024-03-29T10:34:44.729 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQAymQZm9qQmBBAATU/tC/d3or3e9lZRCtr5mw== 2024-03-29T10:34:44.729 INFO:teuthology.orchestra.run.smithi084.stdout:[mgr.x] 2024-03-29T10:34:44.729 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQAlmQZmjf31NRAAy9D23Fmd+0o7mx16IHRzQA== 2024-03-29T10:34:44.729 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.4] 2024-03-29T10:34:44.729 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQA4mQZmw/YxDBAA7TaNGthrpbaYTnMfqOIXsg== 2024-03-29T10:34:44.729 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.5] 2024-03-29T10:34:44.729 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQA7mQZmOF3yChAA7tDvqhagpQwkjjomwr1+YQ== 2024-03-29T10:34:44.729 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.6] 2024-03-29T10:34:44.730 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQA+mQZm/Q1VCRAAbEvZAttduI1XswJZHdkvjQ== 2024-03-29T10:34:44.730 INFO:teuthology.orchestra.run.smithi084.stdout:[osd.7] 2024-03-29T10:34:44.730 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQBBmQZmz6ZNCBAAJHfTFTPdNL0ZxPXz2Ba4kw== 2024-03-29T10:34:44.730 INFO:teuthology.orchestra.run.smithi084.stdout:[client.0] 2024-03-29T10:34:44.730 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQAlmQZmMz+lOBAAYr3LMZ0kYLfrbbClVLGO2g== 2024-03-29T10:34:44.730 INFO:teuthology.orchestra.run.smithi084.stdout:[client.1] 2024-03-29T10:34:44.730 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQAmmQZmFUV0ABAANnFiMBKSeJilXYZ/Z5+CAg== 2024-03-29T10:34:44.752 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr.y] 2024-03-29T10:34:44.752 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQAlmQZmzFbEMRAAK99pTr/W15nhiuzAPEPFEw== 2024-03-29T10:34:44.753 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.0] 2024-03-29T10:34:44.753 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQApmQZmVeGxCRAA9Rb4Y73bYUNdg0Cv3fWBjw== 2024-03-29T10:34:44.753 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.1] 2024-03-29T10:34:44.753 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQAsmQZmIcuKCBAAykp98AqF/jxzl6Rs5UYO9g== 2024-03-29T10:34:44.753 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.2] 2024-03-29T10:34:44.753 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQAvmQZmW8ppBhAA+XEceGs4wGhoRcaySx52eA== 2024-03-29T10:34:44.753 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.3] 2024-03-29T10:34:44.753 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQAymQZm9qQmBBAATU/tC/d3or3e9lZRCtr5mw== 2024-03-29T10:34:44.753 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr.x] 2024-03-29T10:34:44.753 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQAlmQZmjf31NRAAy9D23Fmd+0o7mx16IHRzQA== 2024-03-29T10:34:44.753 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.4] 2024-03-29T10:34:44.753 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQA4mQZmw/YxDBAA7TaNGthrpbaYTnMfqOIXsg== 2024-03-29T10:34:44.753 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.5] 2024-03-29T10:34:44.753 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQA7mQZmOF3yChAA7tDvqhagpQwkjjomwr1+YQ== 2024-03-29T10:34:44.754 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.6] 2024-03-29T10:34:44.754 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQA+mQZm/Q1VCRAAbEvZAttduI1XswJZHdkvjQ== 2024-03-29T10:34:44.754 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.7] 2024-03-29T10:34:44.754 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQBBmQZmz6ZNCBAAJHfTFTPdNL0ZxPXz2Ba4kw== 2024-03-29T10:34:44.754 INFO:teuthology.orchestra.run.smithi086.stdout:[client.0] 2024-03-29T10:34:44.754 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQAlmQZmMz+lOBAAYr3LMZ0kYLfrbbClVLGO2g== 2024-03-29T10:34:44.754 INFO:teuthology.orchestra.run.smithi086.stdout:[client.1] 2024-03-29T10:34:44.754 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQAmmQZmFUV0ABAANnFiMBKSeJilXYZ/Z5+CAg== 2024-03-29T10:34:44.755 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:34:44.773 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:34:44.850 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:44.852 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:44.904 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:44.907 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:44.960 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:44.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:45.014 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:45.016 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:45.067 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:34:45.070 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:34:45.123 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:45.125 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:45.177 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:45.179 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:45.232 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:45.234 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:45.286 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:45.288 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:34:45.340 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:34:45.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:34:45.397 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:34:45.400 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:34:45.458 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T10:34:45.458 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T10:34:45.486 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:34:45.582 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T10:34:45.609 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T10:34:45.675 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:34:45.772 DEBUG:teuthology.orchestra.run.smithi084:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T10:34:45.799 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T10:34:45.826 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:34:45.923 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T10:34:45.950 DEBUG:teuthology.orchestra.run.smithi084:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:34:45.953 DEBUG:teuthology.orchestra.run.smithi086:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:34:46.004 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T10:34:46.005 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T10:34:46.005 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T10:34:46.007 INFO:tasks.ceph.mon.a:Started 2024-03-29T10:34:46.007 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T10:34:46.007 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T10:34:46.009 INFO:tasks.ceph.mon.c:Started 2024-03-29T10:34:46.009 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T10:34:46.010 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T10:34:46.046 INFO:tasks.ceph.mon.b:Started 2024-03-29T10:34:46.047 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T10:34:46.047 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T10:34:46.047 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T10:34:46.049 INFO:tasks.ceph.mgr.y:Started 2024-03-29T10:34:46.049 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T10:34:46.050 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T10:34:46.053 INFO:tasks.ceph.mgr.x:Started 2024-03-29T10:34:46.053 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T10:34:46.053 DEBUG:teuthology.orchestra.run.smithi084:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T10:34:46.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:46.171+0000 7f9457e9a640 10 monclient: get_monmap_and_config 2024-03-29T10:34:46.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:46.171+0000 7f9457e9a640 10 monclient: build_initial_monmap 2024-03-29T10:34:46.156 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:46.171+0000 7f9457e9a640 10 monclient: monmap: 2024-03-29T10:34:46.157 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 0 2024-03-29T10:34:46.157 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:46.157 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-29T10:34:46.172177+0000 2024-03-29T10:34:46.157 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-29T10:34:46.172177+0000 2024-03-29T10:34:46.157 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:46.157 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 1 2024-03-29T10:34:46.157 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:34:46.157 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:34:46.157 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:34:46.157 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T10:34:46.157 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:46.171+0000 7f9457e9a640 1 Processor -- start 2024-03-29T10:34:46.158 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:46.172+0000 7f9457e9a640 1 -- start start 2024-03-29T10:34:46.158 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:46.172+0000 7f9457e9a640 10 monclient: init 2024-03-29T10:34:46.158 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:46.172+0000 7f9457e9a640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:46.158 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:46.172+0000 7f9457e9a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:34:46.159 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:46.172+0000 7f9457e9a640 10 monclient(hunting): picked mon.noname-c con 0x7f945009ac40 addr v1:172.21.15.86:6789/0 2024-03-29T10:34:46.159 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:46.172+0000 7f9457e9a640 10 monclient(hunting): picked mon.noname-b con 0x7f945009c870 addr v1:172.21.15.84:6790/0 2024-03-29T10:34:46.159 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:46.172+0000 7f9457e9a640 10 monclient(hunting): picked mon.noname-a con 0x7f94500b4360 addr v1:172.21.15.84:6789/0 2024-03-29T10:34:46.159 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:46.172+0000 7f9457e9a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f945009dec0 con 0x7f94500b4360 2024-03-29T10:34:46.159 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:46.172+0000 7f9457e9a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f945009da30 con 0x7f945009ac40 2024-03-29T10:34:46.159 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:46.172+0000 7f9457e9a640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94500b39c0 con 0x7f945009c870 2024-03-29T10:34:46.159 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:46.172+0000 7f9457e9a640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:46.159 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:46.172+0000 7f9457e9a640 10 monclient(hunting): authenticate will time out at 788.435746s 2024-03-29T10:34:46.159 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:46.172+0000 7f9455435640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f945009c870 0x7f945013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:36412/0 (socket says 172.21.15.84:36412) 2024-03-29T10:34:46.159 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:46.172+0000 7f9456437640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f94500b4360 0x7f945014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:58120/0 (socket says 172.21.15.84:58120) 2024-03-29T10:34:46.159 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:46.172+0000 7f9455435640 1 -- 172.21.15.84:0/144408868 learned_addr learned my addr 172.21.15.84:0/144408868 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:34:49.158 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:49.173+0000 7f94437fe640 10 monclient(hunting): tick 2024-03-29T10:34:49.158 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:49.173+0000 7f94437fe640 10 monclient(hunting): _check_auth_tickets 2024-03-29T10:34:49.158 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:49.173+0000 7f94437fe640 1 monclient(hunting): continuing hunt 2024-03-29T10:34:49.158 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:49.173+0000 7f94437fe640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T10:34:49.159 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:49.173+0000 7f94437fe640 1 -- 172.21.15.84:0/144408868 >> v1:172.21.15.84:6790/0 conn(0x7f945009c870 legacy=0x7f945013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:49.159 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:49.173+0000 7f94437fe640 1 -- 172.21.15.84:0/144408868 >> v1:172.21.15.86:6789/0 conn(0x7f945009ac40 legacy=0x7f94500b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:49.159 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:49.173+0000 7f94437fe640 1 -- 172.21.15.84:0/144408868 >> v1:172.21.15.84:6789/0 conn(0x7f94500b4360 legacy=0x7f945014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:49.159 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:49.173+0000 7f94437fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:34:49.159 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:49.173+0000 7f94437fe640 10 monclient(hunting): picked mon.noname-c con 0x7f94340036e0 addr v1:172.21.15.86:6789/0 2024-03-29T10:34:49.159 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:49.173+0000 7f94437fe640 10 monclient(hunting): picked mon.noname-b con 0x7f94340072b0 addr v1:172.21.15.84:6790/0 2024-03-29T10:34:49.159 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:49.173+0000 7f94437fe640 10 monclient(hunting): picked mon.noname-a con 0x7f943400af00 addr v1:172.21.15.84:6789/0 2024-03-29T10:34:49.159 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:49.173+0000 7f94437fe640 1 -- 172.21.15.84:0/144408868 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94500b5160 con 0x7f943400af00 2024-03-29T10:34:49.159 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:49.173+0000 7f94437fe640 1 -- 172.21.15.84:0/144408868 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f943400fbd0 con 0x7f94340036e0 2024-03-29T10:34:49.159 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:49.173+0000 7f94437fe640 1 -- 172.21.15.84:0/144408868 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9434010e30 con 0x7f94340072b0 2024-03-29T10:34:49.159 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:49.173+0000 7f94437fe640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:51.340 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.351+0000 7f9454c34640 1 -- 172.21.15.84:0/144408868 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791710685 0 0) 0x7f94500b5160 con 0x7f943400af00 2024-03-29T10:34:51.340 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.351+0000 7f9454c34640 10 monclient(hunting): _negotiate 2024-03-29T10:34:51.341 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.351+0000 7f9454c34640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:51.341 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.351+0000 7f9454c34640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:51.341 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.351+0000 7f9454c34640 10 monclient(hunting): my global_id is 4101 2024-03-29T10:34:51.341 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.351+0000 7f9454c34640 1 -- 172.21.15.84:0/144408868 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f943c003520 con 0x7f943400af00 2024-03-29T10:34:51.341 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.351+0000 7f9454c34640 1 -- 172.21.15.84:0/144408868 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 630147978 0 0) 0x7f943400fbd0 con 0x7f94340036e0 2024-03-29T10:34:51.341 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.351+0000 7f9454c34640 10 monclient(hunting): _negotiate 2024-03-29T10:34:51.341 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.351+0000 7f9454c34640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:51.341 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.351+0000 7f9454c34640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:51.341 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.351+0000 7f9454c34640 10 monclient(hunting): my global_id is 4105 2024-03-29T10:34:51.341 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.351+0000 7f9454c34640 1 -- 172.21.15.84:0/144408868 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94500b5160 con 0x7f94340036e0 2024-03-29T10:34:51.341 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.351+0000 7f9454c34640 1 -- 172.21.15.84:0/144408868 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297802668 0 0) 0x7f9434010e30 con 0x7f94340072b0 2024-03-29T10:34:51.341 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.351+0000 7f9454c34640 10 monclient(hunting): _negotiate 2024-03-29T10:34:51.341 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.351+0000 7f9454c34640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:51.341 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.351+0000 7f9454c34640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:51.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.351+0000 7f9454c34640 10 monclient(hunting): my global_id is 4106 2024-03-29T10:34:51.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.351+0000 7f9454c34640 1 -- 172.21.15.84:0/144408868 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f943400fbd0 con 0x7f94340072b0 2024-03-29T10:34:51.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 1 -- 172.21.15.84:0/144408868 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 757917065 0 0) 0x7f94500b5160 con 0x7f94340036e0 2024-03-29T10:34:51.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 1 -- 172.21.15.84:0/144408868 >> v1:172.21.15.84:6790/0 conn(0x7f94340072b0 legacy=0x7f9434009700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:51.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 1 -- 172.21.15.84:0/144408868 >> v1:172.21.15.84:6789/0 conn(0x7f943400af00 legacy=0x7f943400d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:51.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 10 monclient: _finish_hunting 0 2024-03-29T10:34:51.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 1 monclient: found mon.noname-c 2024-03-29T10:34:51.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:51.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-29T10:34:51.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 1 -- 172.21.15.84:0/144408868 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9434012090 con 0x7f94340036e0 2024-03-29T10:34:51.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 10 monclient: _finish_auth 0 2024-03-29T10:34:51.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 10 monclient: _check_auth_tickets 2024-03-29T10:34:51.342 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:51.343 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 1 -- 172.21.15.84:0/144408868 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f944c001610 con 0x7f94340036e0 2024-03-29T10:34:51.343 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9457e9a640 5 monclient: authenticate success, global_id 4105 2024-03-29T10:34:51.343 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:51.343 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:51.343 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 10 monclient: dump: 2024-03-29T10:34:51.343 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 1 2024-03-29T10:34:51.343 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:51.343 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:51.343 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:51.343 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:51.343 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 1 2024-03-29T10:34:51.343 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:51.343 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:51.343 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:51.344 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T10:34:51.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9457e9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:51.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9455c36640 1 -- 172.21.15.84:0/144408868 >> v1:172.21.15.86:6789/0 conn(0x7f94340036e0 legacy=0x7f9434005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-29T10:34:51.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9455c36640 1 -- 172.21.15.84:0/144408868 >> v1:172.21.15.86:6789/0 conn(0x7f94340036e0 legacy=0x7f9434005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:34:51.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9455c36640 1 --1- 172.21.15.84:0/144408868 >> v1:172.21.15.86:6789/0 conn(0x7f94340036e0 0x7f9434005b10 :-1 s=OPENED pgs=11 cs=1 l=1).handle_message read tag failed 2024-03-29T10:34:51.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9455c36640 1 --1- 172.21.15.84:0/144408868 >> v1:172.21.15.86:6789/0 conn(0x7f94340036e0 0x7f9434005b10 :-1 s=OPENED pgs=11 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:34:51.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 10 monclient: ms_handle_reset current mon v1:172.21.15.86:6789/0 2024-03-29T10:34:51.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:51.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 1 -- 172.21.15.84:0/144408868 >> v1:172.21.15.86:6789/0 conn(0x7f94340036e0 legacy=0x7f9434005b10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:34:51.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:34:51.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 10 monclient(hunting): picked mon.c con 0x7f94340072b0 addr v1:172.21.15.84:6790/0 2024-03-29T10:34:51.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 10 monclient(hunting): picked mon.b con 0x7f943c007960 addr v1:172.21.15.86:6789/0 2024-03-29T10:34:51.344 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 10 monclient(hunting): picked mon.a con 0x7f943400af00 addr v1:172.21.15.84:6789/0 2024-03-29T10:34:51.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 1 -- 172.21.15.84:0/144408868 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f94500b5160 con 0x7f943400af00 2024-03-29T10:34:51.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 1 -- 172.21.15.84:0/144408868 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9434010e30 con 0x7f943c007960 2024-03-29T10:34:51.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 1 -- 172.21.15.84:0/144408868 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f943c010f00 con 0x7f94340072b0 2024-03-29T10:34:51.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:51.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.86:6789/0 2024-03-29T10:34:51.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 1 -- 172.21.15.84:0/144408868 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1137150528 0 0) 0x7f94500b5160 con 0x7f943400af00 2024-03-29T10:34:51.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 10 monclient(hunting): _negotiate 2024-03-29T10:34:51.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:51.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T10:34:51.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.352+0000 7f9454c34640 1 -- 172.21.15.84:0/144408868 --> v1:172.21.15.84:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f943c00fe30 con 0x7f943400af00 2024-03-29T10:34:51.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.353+0000 7f9454c34640 1 -- 172.21.15.84:0/144408868 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 4081103018 0 0) 0x7f943c00fe30 con 0x7f943400af00 2024-03-29T10:34:51.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.353+0000 7f9454c34640 1 -- 172.21.15.84:0/144408868 >> v1:172.21.15.84:6790/0 conn(0x7f94340072b0 legacy=0x7f943c003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:51.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.353+0000 7f9454c34640 1 -- 172.21.15.84:0/144408868 >> v1:172.21.15.86:6789/0 conn(0x7f943c007960 legacy=0x7f943c009d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:51.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.353+0000 7f9454c34640 10 monclient: _finish_hunting 0 2024-03-29T10:34:51.345 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.353+0000 7f9454c34640 1 monclient: found mon.a 2024-03-29T10:34:51.346 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.353+0000 7f9454c34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:51.346 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.353+0000 7f9454c34640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:34:51.346 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.353+0000 7f9454c34640 1 -- 172.21.15.84:0/144408868 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=2+}) v3 -- 0x7f943c012220 con 0x7f943400af00 2024-03-29T10:34:51.346 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.353+0000 7f9454c34640 10 monclient: _finish_auth 0 2024-03-29T10:34:51.346 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.353+0000 7f9454c34640 10 monclient: _check_auth_tickets 2024-03-29T10:34:51.346 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.353+0000 7f9454c34640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:51.346 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.353+0000 7f9454c34640 1 -- 172.21.15.84:0/144408868 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f9448001a30 con 0x7f943400af00 2024-03-29T10:34:51.346 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.353+0000 7f9454c34640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:51.346 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.353+0000 7f9454c34640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:34:51.346 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.353+0000 7f9454c34640 10 monclient: dump: 2024-03-29T10:34:51.346 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 1 2024-03-29T10:34:51.346 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:51.346 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:51.346 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:51.347 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:51.347 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 1 2024-03-29T10:34:51.347 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:51.347 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:51.347 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:51.347 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T10:34:51.347 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.353+0000 7f9457e9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:51.347 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.353+0000 7f9454c34640 1 -- 172.21.15.84:0/144408868 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9448001b90 con 0x7f943400af00 2024-03-29T10:34:51.347 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.353+0000 7f9454c34640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:51.347 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.353+0000 7f9457e9a640 10 monclient: get_monmap_and_config success 2024-03-29T10:34:51.347 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.353+0000 7f9457e9a640 4 set_mon_vals no callback set 2024-03-29T10:34:51.347 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.354+0000 7f9457e9a640 10 monclient: shutdown 2024-03-29T10:34:51.347 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.354+0000 7f9457e9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:51.347 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.354+0000 7f9457e9a640 1 -- 172.21.15.84:0/144408868 >> v1:172.21.15.84:6789/0 conn(0x7f943400af00 legacy=0x7f943c00d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:51.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.354+0000 7f9457e9a640 1 -- 172.21.15.84:0/144408868 shutdown_connections 2024-03-29T10:34:51.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.354+0000 7f9457e9a640 1 -- 172.21.15.84:0/144408868 >> 172.21.15.84:0/144408868 conn(0x7f9450066b00 msgr2=0x7f9450066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:51.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.354+0000 7f9457e9a640 1 -- 172.21.15.84:0/144408868 shutdown_connections 2024-03-29T10:34:51.348 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.354+0000 7f9457e9a640 1 -- 172.21.15.84:0/144408868 wait complete. 2024-03-29T10:34:51.353 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.355+0000 7f9457e9a640 10 monclient: build_initial_monmap 2024-03-29T10:34:51.353 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.355+0000 7f9457e9a640 10 monclient: monmap: 2024-03-29T10:34:51.353 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 0 2024-03-29T10:34:51.353 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:34:51.353 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 0.000000 2024-03-29T10:34:51.354 INFO:teuthology.orchestra.run.smithi084.stderr:created 0.000000 2024-03-29T10:34:51.354 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:51.354 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 1 2024-03-29T10:34:51.354 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:34:51.355 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:34:51.355 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:34:51.355 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T10:34:51.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.355+0000 7f9457e9a640 1 librados: starting msgr at 2024-03-29T10:34:51.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.355+0000 7f9457e9a640 1 librados: starting objecter 2024-03-29T10:34:51.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.355+0000 7f9457e9a640 1 Processor -- start 2024-03-29T10:34:51.355 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.355+0000 7f9457e9a640 1 -- start start 2024-03-29T10:34:51.356 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.355+0000 7f9457e9a640 1 librados: setting wanted keys 2024-03-29T10:34:51.356 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.355+0000 7f9457e9a640 1 librados: calling monclient init 2024-03-29T10:34:51.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.355+0000 7f9457e9a640 10 monclient: init 2024-03-29T10:34:51.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.356+0000 7f9457e9a640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:51.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.356+0000 7f9457e9a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:34:51.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.356+0000 7f9457e9a640 10 monclient(hunting): picked mon.noname-c con 0x7f94340072b0 addr v1:172.21.15.84:6790/0 2024-03-29T10:34:51.357 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.356+0000 7f9457e9a640 10 monclient(hunting): picked mon.noname-b con 0x7f943400af00 addr v1:172.21.15.86:6789/0 2024-03-29T10:34:51.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.356+0000 7f9457e9a640 10 monclient(hunting): picked mon.noname-a con 0x7f943c007960 addr v1:172.21.15.84:6789/0 2024-03-29T10:34:51.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.356+0000 7f9457e9a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9450138a30 con 0x7f943c007960 2024-03-29T10:34:51.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.356+0000 7f9457e9a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94501387d0 con 0x7f943400af00 2024-03-29T10:34:51.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.356+0000 7f9457e9a640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94500b0630 con 0x7f94340072b0 2024-03-29T10:34:51.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.356+0000 7f9457e9a640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:51.358 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.356+0000 7f9457e9a640 10 monclient(hunting): authenticate will time out at 793.620825s 2024-03-29T10:34:51.359 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.356+0000 7f9455c36640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f94340072b0 0x7f94501d9b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:50972/0 (socket says 172.21.15.84:50972) 2024-03-29T10:34:51.359 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.356+0000 7f9455c36640 1 -- 172.21.15.84:0/3124445717 learned_addr learned my addr 172.21.15.84:0/3124445717 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:34:51.359 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.356+0000 7f94427fc640 10 client.?.objecter ms_handle_connect 0x7f94340072b0 2024-03-29T10:34:51.359 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.356+0000 7f94427fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:51.359 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.356+0000 7f94427fc640 10 client.?.objecter ms_handle_connect 0x7f943c007960 2024-03-29T10:34:51.359 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.356+0000 7f94427fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:51.359 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.356+0000 7f94427fc640 10 client.?.objecter ms_handle_connect 0x7f943400af00 2024-03-29T10:34:51.359 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.356+0000 7f94427fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:51.359 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.357+0000 7f94427fc640 1 -- 172.21.15.84:0/3124445717 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050691100 0 0) 0x7f9450138a30 con 0x7f943c007960 2024-03-29T10:34:51.359 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.357+0000 7f94427fc640 10 monclient(hunting): _negotiate 2024-03-29T10:34:51.359 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.357+0000 7f94427fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:51.359 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.357+0000 7f94427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:51.359 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.357+0000 7f94427fc640 10 monclient(hunting): my global_id is 4113 2024-03-29T10:34:51.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.357+0000 7f94427fc640 1 -- 172.21.15.84:0/3124445717 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94340035c0 con 0x7f943c007960 2024-03-29T10:34:51.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.357+0000 7f94427fc640 1 -- 172.21.15.84:0/3124445717 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 340707319 0 0) 0x7f94340035c0 con 0x7f943c007960 2024-03-29T10:34:51.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.357+0000 7f94427fc640 1 -- 172.21.15.84:0/3124445717 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9450138a30 con 0x7f943c007960 2024-03-29T10:34:51.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.357+0000 7f94427fc640 1 -- 172.21.15.84:0/3124445717 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f9448001ea0 con 0x7f943c007960 2024-03-29T10:34:51.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.357+0000 7f94427fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:51.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.357+0000 7f94427fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:34:51.360 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.357+0000 7f94427fc640 10 monclient(hunting): dump: 2024-03-29T10:34:51.360 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 1 2024-03-29T10:34:51.360 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:51.360 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:51.360 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:51.360 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:51.360 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 1 2024-03-29T10:34:51.361 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:51.361 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:51.361 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:51.361 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T10:34:51.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.357+0000 7f94427fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:34:51.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.359+0000 7f94427fc640 1 -- 172.21.15.84:0/3124445717 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3241028600 0 0) 0x7f94500b0630 con 0x7f94340072b0 2024-03-29T10:34:51.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.359+0000 7f94427fc640 10 monclient(hunting): _negotiate 2024-03-29T10:34:51.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.359+0000 7f94427fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:51.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.359+0000 7f94427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:51.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.359+0000 7f94427fc640 10 monclient(hunting): my global_id is 4109 2024-03-29T10:34:51.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.359+0000 7f94427fc640 1 -- 172.21.15.84:0/3124445717 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94340035c0 con 0x7f94340072b0 2024-03-29T10:34:51.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.359+0000 7f9455c36640 1 -- 172.21.15.84:0/3124445717 >> v1:172.21.15.84:6790/0 conn(0x7f94340072b0 legacy=0x7f94501d9b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-29T10:34:51.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.359+0000 7f9455c36640 1 -- 172.21.15.84:0/3124445717 >> v1:172.21.15.84:6790/0 conn(0x7f94340072b0 legacy=0x7f94501d9b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:34:51.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.359+0000 7f9455c36640 1 --1- 172.21.15.84:0/3124445717 >> v1:172.21.15.84:6790/0 conn(0x7f94340072b0 0x7f94501d9b10 :-1 s=OPENED pgs=19 cs=1 l=1).handle_message read tag failed 2024-03-29T10:34:51.361 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.359+0000 7f9455c36640 1 --1- 172.21.15.84:0/3124445717 >> v1:172.21.15.84:6790/0 conn(0x7f94340072b0 0x7f94501d9b10 :-1 s=OPENED pgs=19 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:34:51.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.359+0000 7f94427fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.84:6790/0 2024-03-29T10:34:51.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.359+0000 7f94427fc640 1 -- 172.21.15.84:0/3124445717 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3001645286 0 0) 0x7f9450138a30 con 0x7f943c007960 2024-03-29T10:34:51.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.359+0000 7f94427fc640 1 -- 172.21.15.84:0/3124445717 >> v1:172.21.15.84:6790/0 conn(0x7f94340072b0 legacy=0x7f94501d9b10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:34:51.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.359+0000 7f94427fc640 1 -- 172.21.15.84:0/3124445717 >> v1:172.21.15.86:6789/0 conn(0x7f943400af00 legacy=0x7f94501379b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:51.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f94427fc640 10 monclient: _finish_hunting 0 2024-03-29T10:34:51.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f94427fc640 1 monclient: found mon.a 2024-03-29T10:34:51.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f94427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:51.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f94427fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:34:51.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f94427fc640 1 -- 172.21.15.84:0/3124445717 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94500b1210 con 0x7f943c007960 2024-03-29T10:34:51.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f94427fc640 10 monclient: _finish_auth 0 2024-03-29T10:34:51.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f94427fc640 10 monclient: _check_auth_tickets 2024-03-29T10:34:51.362 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f94427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:51.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f9457e9a640 5 monclient: authenticate success, global_id 4113 2024-03-29T10:34:51.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f9457e9a640 10 monclient: _renew_subs 2024-03-29T10:34:51.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f9457e9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:34:51.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f9457e9a640 1 -- 172.21.15.84:0/3124445717 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94500b0d90 con 0x7f943c007960 2024-03-29T10:34:51.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f9457e9a640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:34:51.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f9457e9a640 10 monclient: _renew_subs 2024-03-29T10:34:51.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f9457e9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:34:51.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f9457e9a640 1 -- 172.21.15.84:0/3124445717 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9450138dc0 con 0x7f943c007960 2024-03-29T10:34:51.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f9457e9a640 1 librados: init done 2024-03-29T10:34:51.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f94427fc640 1 -- 172.21.15.84:0/3124445717 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9448002000 con 0x7f943c007960 2024-03-29T10:34:51.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f94427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:51.363 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f94427fc640 1 -- 172.21.15.84:0/3124445717 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f94480050b0 con 0x7f943c007960 2024-03-29T10:34:51.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f94427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:51.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f94427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:34:51.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f9443fff640 4 set_mon_vals no callback set 2024-03-29T10:34:51.364 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f94427fc640 10 monclient: dump: 2024-03-29T10:34:51.364 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 1 2024-03-29T10:34:51.364 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:51.364 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:51.364 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:51.364 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:51.364 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 1 2024-03-29T10:34:51.364 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:51.364 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:51.364 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:51.364 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T10:34:51.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f94427fc640 1 -- 172.21.15.84:0/3124445717 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9448005660 con 0x7f943c007960 2024-03-29T10:34:51.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f94427fc640 1 -- 172.21.15.84:0/3124445717 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2637426337 0 0) 0x7f9448006b80 con 0x7f943c007960 2024-03-29T10:34:51.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f94427fc640 10 client.4113.objecter ms_dispatch 0x7f9450000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T10:34:51.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f94427fc640 3 client.4113.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T10:34:51.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f94427fc640 3 client.4113.objecter handle_osd_map decoding full epoch 1 2024-03-29T10:34:51.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.360+0000 7f94427fc640 20 client.4113.objecter dump_active .. 0 homeless 2024-03-29T10:34:51.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.361+0000 7f9457e9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:51.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.361+0000 7f9457e9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:51.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.361+0000 7f9457e9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:34:51.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.361+0000 7f9457e9a640 1 -- 172.21.15.84:0/3124445717 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9438005080 con 0x7f943c007960 2024-03-29T10:34:51.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.363+0000 7f94427fc640 1 -- 172.21.15.84:0/3124445717 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9448005af0 con 0x7f943c007960 2024-03-29T10:34:51.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.363+0000 7f94427fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:51.365 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.363+0000 7f94427fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:34:51.420 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.434+0000 7f9457e9a640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:34:51.420 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.434+0000 7f9457e9a640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:34:51.421 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.434+0000 7f9457e9a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:34:51.421 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.434+0000 7f9457e9a640 1 -- 172.21.15.84:0/3124445717 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f9438005330 con 0x7f943c007960 2024-03-29T10:34:51.424 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.438+0000 7f94427fc640 1 -- 172.21.15.84:0/3124445717 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f9448005d70 con 0x7f943c007960 2024-03-29T10:34:51.424 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.438+0000 7f94427fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:34:51.424 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.438+0000 7f94427fc640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T10:34:51.424 INFO:teuthology.orchestra.run.smithi084.stderr:adjusted tunables profile to default 2024-03-29T10:34:51.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.439+0000 7f9457e9a640 10 librados: watch_flush enter 2024-03-29T10:34:51.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.439+0000 7f9457e9a640 10 librados: watch_flush exit 2024-03-29T10:34:51.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.439+0000 7f9457e9a640 20 client.4113.objecter shutdown clearing up homeless session... 2024-03-29T10:34:51.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.439+0000 7f9457e9a640 10 client.4113.objecter successfully canceled tick 2024-03-29T10:34:51.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.439+0000 7f9457e9a640 10 monclient: shutdown 2024-03-29T10:34:51.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.439+0000 7f9457e9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:51.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.439+0000 7f9457e9a640 1 -- 172.21.15.84:0/3124445717 >> v1:172.21.15.84:6789/0 conn(0x7f943c007960 legacy=0x7f94501380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:51.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.439+0000 7f9457e9a640 1 -- 172.21.15.84:0/3124445717 shutdown_connections 2024-03-29T10:34:51.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.439+0000 7f9457e9a640 1 -- 172.21.15.84:0/3124445717 >> 172.21.15.84:0/3124445717 conn(0x7f9450066b00 msgr2=0x7f94500b7c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:51.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.439+0000 7f9457e9a640 1 -- 172.21.15.84:0/3124445717 shutdown_connections 2024-03-29T10:34:51.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.439+0000 7f9457e9a640 1 -- 172.21.15.84:0/3124445717 wait complete. 2024-03-29T10:34:51.426 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:51.439+0000 7f9457e9a640 1 librados: shutdown 2024-03-29T10:34:51.436 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T10:34:51.436 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T10:34:51.437 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T10:34:51.437 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T10:34:51.464 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T10:34:51.464 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T10:34:51.531 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T10:34:51.531 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T10:34:51.598 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-03-29T10:34:51.598 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T10:34:51.665 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T10:34:51.665 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T10:34:51.692 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T10:34:51.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T10:34:51.759 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T10:34:51.759 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T10:34:51.825 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-29T10:34:51.840+0000 7f4e58c21040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:34:51.827 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T10:34:51.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T10:34:51.843 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T10:34:51.857+0000 7f187bc42040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:34:51.881 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-29T10:34:51.896+0000 7f4e58c21040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:34:51.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 8652d97b-41e0-4025-a2e9-9c78b097ffb4 0 2024-03-29T10:34:51.896 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T10:34:51.910+0000 7f187bc42040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:34:52.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.047+0000 7f6c45e0f640 10 monclient: get_monmap_and_config 2024-03-29T10:34:52.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.047+0000 7f6c45e0f640 10 monclient: build_initial_monmap 2024-03-29T10:34:52.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.047+0000 7f6c45e0f640 10 monclient: monmap: 2024-03-29T10:34:52.033 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-29T10:34:52.033 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.033 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:52.048620+0000 2024-03-29T10:34:52.033 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:52.048620+0000 2024-03-29T10:34:52.033 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:52.033 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.033 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:34:52.033 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:34:52.033 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:34:52.033 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.048+0000 7f6c45e0f640 1 Processor -- start 2024-03-29T10:34:52.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.048+0000 7f6c45e0f640 1 -- start start 2024-03-29T10:34:52.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.048+0000 7f6c45e0f640 10 monclient: init 2024-03-29T10:34:52.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.048+0000 7f6c45e0f640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:52.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.048+0000 7f6c45e0f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:34:52.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.048+0000 7f6c45e0f640 10 monclient(hunting): picked mon.noname-a con 0x7f6c4009ac40 addr v1:172.21.15.84:6789/0 2024-03-29T10:34:52.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.048+0000 7f6c45e0f640 10 monclient(hunting): picked mon.noname-c con 0x7f6c4009c870 addr v1:172.21.15.86:6789/0 2024-03-29T10:34:52.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c45e0f640 10 monclient(hunting): picked mon.noname-b con 0x7f6c400b4360 addr v1:172.21.15.84:6790/0 2024-03-29T10:34:52.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c45e0f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c4009dec0 con 0x7f6c4009ac40 2024-03-29T10:34:52.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c45e0f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c4009da30 con 0x7f6c4009c870 2024-03-29T10:34:52.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c45e0f640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c400b39c0 con 0x7f6c400b4360 2024-03-29T10:34:52.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c45e0f640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:52.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c45e0f640 10 monclient(hunting): authenticate will time out at 776.214172s 2024-03-29T10:34:52.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3effd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f6c4009c870 0x7f6c4013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:50266/0 (socket says 172.21.15.86:50266) 2024-03-29T10:34:52.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3effd640 1 -- 172.21.15.86:0/3591181862 learned_addr learned my addr 172.21.15.86:0/3591181862 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T10:34:52.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 1 -- 172.21.15.86:0/3591181862 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2971216996 0 0) 0x7f6c4009da30 con 0x7f6c4009c870 2024-03-29T10:34:52.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:34:52.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:52.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:52.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 10 monclient(hunting): my global_id is 4108 2024-03-29T10:34:52.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 1 -- 172.21.15.86:0/3591181862 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c2c003520 con 0x7f6c4009c870 2024-03-29T10:34:52.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 1 -- 172.21.15.86:0/3591181862 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648962994 0 0) 0x7f6c28001110 con 0x7f6c4009ac40 2024-03-29T10:34:52.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:34:52.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:52.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:52.046 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 10 monclient(hunting): my global_id is 4116 2024-03-29T10:34:52.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 1 -- 172.21.15.86:0/3591181862 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c4009da30 con 0x7f6c4009ac40 2024-03-29T10:34:52.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 1 -- 172.21.15.86:0/3591181862 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4120561709 0 0) 0x7f6c400b39c0 con 0x7f6c400b4360 2024-03-29T10:34:52.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:34:52.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:52.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:52.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 10 monclient(hunting): my global_id is 4112 2024-03-29T10:34:52.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 1 -- 172.21.15.86:0/3591181862 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c28001110 con 0x7f6c400b4360 2024-03-29T10:34:52.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 1 -- 172.21.15.86:0/3591181862 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2457787334 0 0) 0x7f6c2c003520 con 0x7f6c4009c870 2024-03-29T10:34:52.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 1 -- 172.21.15.86:0/3591181862 >> v1:172.21.15.84:6790/0 conn(0x7f6c400b4360 legacy=0x7f6c4014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 1 -- 172.21.15.86:0/3591181862 >> v1:172.21.15.84:6789/0 conn(0x7f6c4009ac40 legacy=0x7f6c400b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:34:52.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 1 monclient: found mon.noname-c 2024-03-29T10:34:52.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:52.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-29T10:34:52.047 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 1 -- 172.21.15.86:0/3591181862 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c400b5160 con 0x7f6c4009c870 2024-03-29T10:34:52.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:34:52.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:34:52.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:52.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 1 -- 172.21.15.86:0/3591181862 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f6c38001690 con 0x7f6c4009c870 2024-03-29T10:34:52.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c45e0f640 5 monclient: authenticate success, global_id 4108 2024-03-29T10:34:52.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c45e0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:52.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:52.048 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 10 monclient: dump: 2024-03-29T10:34:52.048 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.048 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.048 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.048 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.048 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.049 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.049 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.049 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.049 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.049 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c45e0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:52.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 1 -- 172.21.15.86:0/3591181862 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c38001870 con 0x7f6c4009c870 2024-03-29T10:34:52.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:52.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 1 -- 172.21.15.86:0/3591181862 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f6c38001c80 con 0x7f6c4009c870 2024-03-29T10:34:52.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c45e0f640 10 monclient: get_monmap_and_config success 2024-03-29T10:34:52.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c45e0f640 4 set_mon_vals no callback set 2024-03-29T10:34:52.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.049 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:52.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.049+0000 7f6c3e7fc640 10 monclient: dump: 2024-03-29T10:34:52.050 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.050 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.050 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.050 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.050 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.050 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.050 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.050 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.050 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.050 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.050+0000 7f6c45e0f640 10 monclient: shutdown 2024-03-29T10:34:52.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.050+0000 7f6c45e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:52.050 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.050+0000 7f6c45e0f640 1 -- 172.21.15.86:0/3591181862 >> v1:172.21.15.86:6789/0 conn(0x7f6c4009c870 legacy=0x7f6c4013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.050+0000 7f6c45e0f640 1 -- 172.21.15.86:0/3591181862 shutdown_connections 2024-03-29T10:34:52.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.050+0000 7f6c45e0f640 1 -- 172.21.15.86:0/3591181862 >> 172.21.15.86:0/3591181862 conn(0x7f6c40066b00 msgr2=0x7f6c40066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:52.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.050+0000 7f6c45e0f640 1 -- 172.21.15.86:0/3591181862 shutdown_connections 2024-03-29T10:34:52.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.050+0000 7f6c45e0f640 1 -- 172.21.15.86:0/3591181862 wait complete. 2024-03-29T10:34:52.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.051+0000 7f6c45e0f640 10 monclient: build_initial_monmap 2024-03-29T10:34:52.051 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.051+0000 7f6c45e0f640 10 monclient: monmap: 2024-03-29T10:34:52.051 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-29T10:34:52.051 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:34:52.051 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-03-29T10:34:52.051 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-03-29T10:34:52.051 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:52.051 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.051 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:34:52.052 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:34:52.052 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:34:52.052 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.051+0000 7f6c45e0f640 1 librados: starting msgr at 2024-03-29T10:34:52.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.051+0000 7f6c45e0f640 1 librados: starting objecter 2024-03-29T10:34:52.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.052+0000 7f6c45e0f640 1 Processor -- start 2024-03-29T10:34:52.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.052+0000 7f6c45e0f640 1 -- start start 2024-03-29T10:34:52.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.052+0000 7f6c45e0f640 1 librados: setting wanted keys 2024-03-29T10:34:52.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.052+0000 7f6c45e0f640 1 librados: calling monclient init 2024-03-29T10:34:52.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.052+0000 7f6c45e0f640 10 monclient: init 2024-03-29T10:34:52.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c45e0f640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:52.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c45e0f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:34:52.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c45e0f640 10 monclient(hunting): picked mon.noname-c con 0x7f6c4009ac40 addr v1:172.21.15.84:6790/0 2024-03-29T10:34:52.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c45e0f640 10 monclient(hunting): picked mon.noname-b con 0x7f6c40138df0 addr v1:172.21.15.86:6789/0 2024-03-29T10:34:52.052 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c45e0f640 10 monclient(hunting): picked mon.noname-a con 0x7f6c400b05c0 addr v1:172.21.15.84:6789/0 2024-03-29T10:34:52.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c45e0f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c400afe90 con 0x7f6c400b05c0 2024-03-29T10:34:52.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c45e0f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c400b1920 con 0x7f6c40138df0 2024-03-29T10:34:52.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c45e0f640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c400b0a10 con 0x7f6c4009ac40 2024-03-29T10:34:52.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c45e0f640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:52.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c45e0f640 10 monclient(hunting): authenticate will time out at 776.218332s 2024-03-29T10:34:52.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c3effd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f6c40138df0 0x7f6c400af780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:50270/0 (socket says 172.21.15.86:50270) 2024-03-29T10:34:52.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c3effd640 1 -- 172.21.15.86:0/3357353441 learned_addr learned my addr 172.21.15.86:0/3357353441 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T10:34:52.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 10 client.?.objecter ms_handle_connect 0x7f6c40138df0 2024-03-29T10:34:52.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:52.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 10 client.?.objecter ms_handle_connect 0x7f6c400b05c0 2024-03-29T10:34:52.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:52.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 10 client.?.objecter ms_handle_connect 0x7f6c4009ac40 2024-03-29T10:34:52.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:52.053 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 1 -- 172.21.15.86:0/3357353441 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877759464 0 0) 0x7f6c400b1920 con 0x7f6c40138df0 2024-03-29T10:34:52.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 10 monclient(hunting): _negotiate 2024-03-29T10:34:52.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:52.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:52.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 10 monclient(hunting): my global_id is 4111 2024-03-29T10:34:52.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 1 -- 172.21.15.86:0/3357353441 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c20004eb0 con 0x7f6c40138df0 2024-03-29T10:34:52.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 1 -- 172.21.15.86:0/3357353441 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3132778878 0 0) 0x7f6c400afe90 con 0x7f6c400b05c0 2024-03-29T10:34:52.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 10 monclient(hunting): _negotiate 2024-03-29T10:34:52.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:52.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:52.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 10 monclient(hunting): my global_id is 4119 2024-03-29T10:34:52.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 1 -- 172.21.15.86:0/3357353441 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c400b1920 con 0x7f6c400b05c0 2024-03-29T10:34:52.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 1 -- 172.21.15.86:0/3357353441 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2243125846 0 0) 0x7f6c400b0a10 con 0x7f6c4009ac40 2024-03-29T10:34:52.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 10 monclient(hunting): _negotiate 2024-03-29T10:34:52.054 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:52.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:52.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 10 monclient(hunting): my global_id is 4115 2024-03-29T10:34:52.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 1 -- 172.21.15.86:0/3357353441 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c400afe90 con 0x7f6c4009ac40 2024-03-29T10:34:52.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 1 -- 172.21.15.86:0/3357353441 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 504848738 0 0) 0x7f6c400b1920 con 0x7f6c400b05c0 2024-03-29T10:34:52.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 1 -- 172.21.15.86:0/3357353441 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c400b0a10 con 0x7f6c400b05c0 2024-03-29T10:34:52.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 1 -- 172.21.15.86:0/3357353441 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1005092642 0 0) 0x7f6c20004eb0 con 0x7f6c40138df0 2024-03-29T10:34:52.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 1 -- 172.21.15.86:0/3357353441 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c400b1920 con 0x7f6c40138df0 2024-03-29T10:34:52.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 1 -- 172.21.15.86:0/3357353441 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f6c30001b50 con 0x7f6c400b05c0 2024-03-29T10:34:52.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:34:52.055 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 10 monclient(hunting): dump: 2024-03-29T10:34:52.055 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.055 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.055 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.056 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.056 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.056 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.056 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.056 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.056 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.056 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:34:52.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 1 -- 172.21.15.86:0/3357353441 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f6c38001bc0 con 0x7f6c40138df0 2024-03-29T10:34:52.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:52.056 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 10 monclient(hunting): dump: 2024-03-29T10:34:52.056 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.056 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.057 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.057 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.057 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.057 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.057 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.057 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.057 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.057 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 1 -- 172.21.15.86:0/3357353441 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1024274825 0 0) 0x7f6c400afe90 con 0x7f6c4009ac40 2024-03-29T10:34:52.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 1 -- 172.21.15.86:0/3357353441 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c20004eb0 con 0x7f6c4009ac40 2024-03-29T10:34:52.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 1 -- 172.21.15.86:0/3357353441 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f6c280018a0 con 0x7f6c4009ac40 2024-03-29T10:34:52.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:34:52.057 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 10 monclient(hunting): dump: 2024-03-29T10:34:52.058 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.058 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.058 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.058 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.058 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.058 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.058 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.058 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.058 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.058 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.053+0000 7f6c37fff640 1 -- 172.21.15.86:0/3357353441 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 493452427 0 0) 0x7f6c400b0a10 con 0x7f6c400b05c0 2024-03-29T10:34:52.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c37fff640 1 -- 172.21.15.86:0/3357353441 >> v1:172.21.15.84:6790/0 conn(0x7f6c4009ac40 legacy=0x7f6c401d7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c37fff640 1 -- 172.21.15.86:0/3357353441 >> v1:172.21.15.86:6789/0 conn(0x7f6c40138df0 legacy=0x7f6c400af780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.058 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c37fff640 10 monclient: _finish_hunting 0 2024-03-29T10:34:52.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c37fff640 1 monclient: found mon.a 2024-03-29T10:34:52.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c37fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:52.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c37fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:34:52.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c37fff640 1 -- 172.21.15.86:0/3357353441 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c400b00e0 con 0x7f6c400b05c0 2024-03-29T10:34:52.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c37fff640 10 monclient: _finish_auth 0 2024-03-29T10:34:52.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c37fff640 10 monclient: _check_auth_tickets 2024-03-29T10:34:52.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:52.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c45e0f640 5 monclient: authenticate success, global_id 4119 2024-03-29T10:34:52.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c45e0f640 10 monclient: _renew_subs 2024-03-29T10:34:52.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c45e0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:34:52.059 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c45e0f640 1 -- 172.21.15.86:0/3357353441 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c40139890 con 0x7f6c400b05c0 2024-03-29T10:34:52.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c45e0f640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:34:52.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c45e0f640 10 monclient: _renew_subs 2024-03-29T10:34:52.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c45e0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:34:52.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c45e0f640 1 -- 172.21.15.86:0/3357353441 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c401389f0 con 0x7f6c400b05c0 2024-03-29T10:34:52.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c45e0f640 1 librados: init done 2024-03-29T10:34:52.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c37fff640 1 -- 172.21.15.86:0/3357353441 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c300020a0 con 0x7f6c400b05c0 2024-03-29T10:34:52.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c37fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:52.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c37fff640 1 -- 172.21.15.86:0/3357353441 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f6c30003b30 con 0x7f6c400b05c0 2024-03-29T10:34:52.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c37fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:34:52.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c3dffb640 4 set_mon_vals no callback set 2024-03-29T10:34:52.060 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c37fff640 10 monclient: dump: 2024-03-29T10:34:52.060 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.061 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.061 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.061 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.061 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.061 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.061 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.061 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.061 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.061 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c37fff640 1 -- 172.21.15.86:0/3357353441 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6c30004020 con 0x7f6c400b05c0 2024-03-29T10:34:52.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c37fff640 1 -- 172.21.15.86:0/3357353441 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 4237797687 0 0) 0x7f6c300055b0 con 0x7f6c400b05c0 2024-03-29T10:34:52.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c37fff640 10 client.4119.objecter ms_dispatch 0x7f6c40000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T10:34:52.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c37fff640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T10:34:52.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c37fff640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-29T10:34:52.061 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c37fff640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-29T10:34:52.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c45e0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:52.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c45e0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:52.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c45e0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:34:52.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.054+0000 7f6c45e0f640 1 -- 172.21.15.86:0/3357353441 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c24005080 con 0x7f6c400b05c0 2024-03-29T10:34:52.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.057+0000 7f6c37fff640 1 -- 172.21.15.86:0/3357353441 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6c30005890 con 0x7f6c400b05c0 2024-03-29T10:34:52.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.057+0000 7f6c37fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:52.062 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.057+0000 7f6c37fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:34:52.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.127+0000 7f6c45e0f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8652d97b-41e0-4025-a2e9-9c78b097ffb4", "id": 0}] 2024-03-29T10:34:52.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.127+0000 7f6c45e0f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8652d97b-41e0-4025-a2e9-9c78b097ffb4", "id": 0}] 2024-03-29T10:34:52.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.127+0000 7f6c45e0f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:34:52.112 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.127+0000 7f6c45e0f640 1 -- 172.21.15.86:0/3357353441 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd new", "uuid": "8652d97b-41e0-4025-a2e9-9c78b097ffb4", "id": 0} v 0) v1 -- 0x7f6c240058b0 con 0x7f6c400b05c0 2024-03-29T10:34:52.113 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.128+0000 7f6c37fff640 1 -- 172.21.15.86:0/3357353441 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8652d97b-41e0-4025-a2e9-9c78b097ffb4", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 2877097663 0 732707515) 0x7f6c30005a30 con 0x7f6c400b05c0 2024-03-29T10:34:52.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.128+0000 7f6c37fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8652d97b-41e0-4025-a2e9-9c78b097ffb4", "id": 0}] 2024-03-29T10:34:52.114 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.128+0000 7f6c37fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:34:52.114 INFO:teuthology.orchestra.run.smithi086.stdout:0 2024-03-29T10:34:52.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.129+0000 7f6c45e0f640 10 librados: watch_flush enter 2024-03-29T10:34:52.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.129+0000 7f6c45e0f640 10 librados: watch_flush exit 2024-03-29T10:34:52.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.129+0000 7f6c45e0f640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-29T10:34:52.115 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.129+0000 7f6c45e0f640 10 client.4119.objecter successfully canceled tick 2024-03-29T10:34:52.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.129+0000 7f6c45e0f640 10 monclient: shutdown 2024-03-29T10:34:52.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.129+0000 7f6c45e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:52.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.129+0000 7f6c45e0f640 1 -- 172.21.15.86:0/3357353441 >> v1:172.21.15.84:6789/0 conn(0x7f6c400b05c0 legacy=0x7f6c40138100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.129+0000 7f6c45e0f640 1 -- 172.21.15.86:0/3357353441 shutdown_connections 2024-03-29T10:34:52.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.129+0000 7f6c45e0f640 1 -- 172.21.15.86:0/3357353441 >> 172.21.15.86:0/3357353441 conn(0x7f6c40066b00 msgr2=0x7f6c4013d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:52.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.129+0000 7f6c45e0f640 1 -- 172.21.15.86:0/3357353441 shutdown_connections 2024-03-29T10:34:52.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.129+0000 7f6c45e0f640 1 -- 172.21.15.86:0/3357353441 wait complete. 2024-03-29T10:34:52.116 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.130+0000 7f6c45e0f640 1 librados: shutdown 2024-03-29T10:34:52.125 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 81f7bc3e-3d25-4774-b43b-8eddffd49bfd 1 2024-03-29T10:34:52.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.240+0000 7ff1f72ff640 10 monclient: get_monmap_and_config 2024-03-29T10:34:52.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.240+0000 7ff1f72ff640 10 monclient: build_initial_monmap 2024-03-29T10:34:52.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.240+0000 7ff1f72ff640 10 monclient: monmap: 2024-03-29T10:34:52.226 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-29T10:34:52.226 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.226 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:52.241432+0000 2024-03-29T10:34:52.227 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:52.241432+0000 2024-03-29T10:34:52.227 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:52.227 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.227 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:34:52.227 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:34:52.227 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:34:52.227 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.240+0000 7ff1f72ff640 1 Processor -- start 2024-03-29T10:34:52.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.241+0000 7ff1f72ff640 1 -- start start 2024-03-29T10:34:52.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.241+0000 7ff1f72ff640 10 monclient: init 2024-03-29T10:34:52.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.241+0000 7ff1f72ff640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:52.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.241+0000 7ff1f72ff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:34:52.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.241+0000 7ff1f72ff640 10 monclient(hunting): picked mon.noname-c con 0x7ff1f00af360 addr v1:172.21.15.86:6789/0 2024-03-29T10:34:52.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.241+0000 7ff1f72ff640 10 monclient(hunting): picked mon.noname-a con 0x7ff1f006af60 addr v1:172.21.15.84:6789/0 2024-03-29T10:34:52.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.241+0000 7ff1f72ff640 10 monclient(hunting): picked mon.noname-b con 0x7ff1f0145e60 addr v1:172.21.15.84:6790/0 2024-03-29T10:34:52.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.241+0000 7ff1f72ff640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f00b0520 con 0x7ff1f006af60 2024-03-29T10:34:52.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.241+0000 7ff1f72ff640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f00aff20 con 0x7ff1f00af360 2024-03-29T10:34:52.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.241+0000 7ff1f72ff640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f00b1da0 con 0x7ff1f0145e60 2024-03-29T10:34:52.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.241+0000 7ff1f72ff640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:52.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.241+0000 7ff1f72ff640 10 monclient(hunting): authenticate will time out at 776.406917s 2024-03-29T10:34:52.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.241+0000 7ff1f509b640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7ff1f00af360 0x7ff1f006c320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:50276/0 (socket says 172.21.15.86:50276) 2024-03-29T10:34:52.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.241+0000 7ff1f509b640 1 -- 172.21.15.86:0/1069158237 learned_addr learned my addr 172.21.15.86:0/1069158237 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T10:34:52.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1e7fff640 1 -- 172.21.15.86:0/1069158237 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 976280000 0 0) 0x7ff1f00aff20 con 0x7ff1f00af360 2024-03-29T10:34:52.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1e7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:34:52.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:52.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:52.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1e7fff640 10 monclient(hunting): my global_id is 4114 2024-03-29T10:34:52.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1e7fff640 1 -- 172.21.15.86:0/1069158237 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1dc003520 con 0x7ff1f00af360 2024-03-29T10:34:52.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1e7fff640 1 -- 172.21.15.86:0/1069158237 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1041673667 0 0) 0x7ff1dc003520 con 0x7ff1f00af360 2024-03-29T10:34:52.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1e7fff640 1 -- 172.21.15.86:0/1069158237 >> v1:172.21.15.84:6790/0 conn(0x7ff1f0145e60 legacy=0x7ff1f014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1e7fff640 1 -- 172.21.15.86:0/1069158237 >> v1:172.21.15.84:6789/0 conn(0x7ff1f006af60 legacy=0x7ff1f0145750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1e7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:34:52.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1e7fff640 1 monclient: found mon.noname-c 2024-03-29T10:34:52.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:52.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1e7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-29T10:34:52.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1e7fff640 1 -- 172.21.15.86:0/1069158237 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f00b2080 con 0x7ff1f00af360 2024-03-29T10:34:52.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1e7fff640 10 monclient: _finish_auth 0 2024-03-29T10:34:52.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1e7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:34:52.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:52.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1f72ff640 5 monclient: authenticate success, global_id 4114 2024-03-29T10:34:52.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1e7fff640 1 -- 172.21.15.86:0/1069158237 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7ff1ec001690 con 0x7ff1f00af360 2024-03-29T10:34:52.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1f72ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:52.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1e7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:52.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1e7fff640 10 monclient: dump: 2024-03-29T10:34:52.237 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.237 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.238 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.238 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.238 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.238 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.238 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.238 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.238 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.238 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1f72ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:52.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1e7fff640 1 -- 172.21.15.86:0/1069158237 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1ec001870 con 0x7ff1f00af360 2024-03-29T10:34:52.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1e7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:52.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1e7fff640 1 -- 172.21.15.86:0/1069158237 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7ff1ec001c00 con 0x7ff1f00af360 2024-03-29T10:34:52.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1f72ff640 10 monclient: get_monmap_and_config success 2024-03-29T10:34:52.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1f72ff640 4 set_mon_vals no callback set 2024-03-29T10:34:52.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1e7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:52.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1e7fff640 10 monclient: dump: 2024-03-29T10:34:52.239 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.239 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.239 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.239 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.239 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.239 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.239 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.239 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.239 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.239 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1f72ff640 10 monclient: shutdown 2024-03-29T10:34:52.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1f72ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:52.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1f72ff640 1 -- 172.21.15.86:0/1069158237 >> v1:172.21.15.86:6789/0 conn(0x7ff1f00af360 legacy=0x7ff1f006c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1f72ff640 1 -- 172.21.15.86:0/1069158237 shutdown_connections 2024-03-29T10:34:52.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.242+0000 7ff1f72ff640 1 -- 172.21.15.86:0/1069158237 >> 172.21.15.86:0/1069158237 conn(0x7ff1f00b58f0 msgr2=0x7ff1f00b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:52.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.243+0000 7ff1f72ff640 1 -- 172.21.15.86:0/1069158237 shutdown_connections 2024-03-29T10:34:52.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.243+0000 7ff1f72ff640 1 -- 172.21.15.86:0/1069158237 wait complete. 2024-03-29T10:34:52.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.243+0000 7ff1f72ff640 10 monclient: build_initial_monmap 2024-03-29T10:34:52.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.243+0000 7ff1f72ff640 10 monclient: monmap: 2024-03-29T10:34:52.240 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-29T10:34:52.240 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:34:52.240 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-03-29T10:34:52.240 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-03-29T10:34:52.240 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:52.240 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.240 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:34:52.240 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:34:52.240 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:34:52.240 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.243+0000 7ff1f72ff640 1 librados: starting msgr at 2024-03-29T10:34:52.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.243+0000 7ff1f72ff640 1 librados: starting objecter 2024-03-29T10:34:52.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.243+0000 7ff1f72ff640 1 Processor -- start 2024-03-29T10:34:52.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1f72ff640 1 -- start start 2024-03-29T10:34:52.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1f72ff640 1 librados: setting wanted keys 2024-03-29T10:34:52.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1f72ff640 1 librados: calling monclient init 2024-03-29T10:34:52.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1f72ff640 10 monclient: init 2024-03-29T10:34:52.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1f72ff640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:52.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1f72ff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:34:52.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1f72ff640 10 monclient(hunting): picked mon.noname-c con 0x7ff1f006af60 addr v1:172.21.15.84:6790/0 2024-03-29T10:34:52.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1f72ff640 10 monclient(hunting): picked mon.noname-a con 0x7ff1f00af360 addr v1:172.21.15.84:6789/0 2024-03-29T10:34:52.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1f72ff640 10 monclient(hunting): picked mon.noname-b con 0x7ff1f0145e60 addr v1:172.21.15.86:6789/0 2024-03-29T10:34:52.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1f72ff640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f01413d0 con 0x7ff1f00af360 2024-03-29T10:34:52.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1f72ff640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f013b430 con 0x7ff1f0145e60 2024-03-29T10:34:52.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1f72ff640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f013b5c0 con 0x7ff1f006af60 2024-03-29T10:34:52.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1f72ff640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:52.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1f72ff640 10 monclient(hunting): authenticate will time out at 776.409667s 2024-03-29T10:34:52.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1f589c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7ff1f0145e60 0x7ff1f01daa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:50282/0 (socket says 172.21.15.86:50282) 2024-03-29T10:34:52.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1f589c640 1 -- 172.21.15.86:0/890627672 learned_addr learned my addr 172.21.15.86:0/890627672 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T10:34:52.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1e5ffb640 10 client.?.objecter ms_handle_connect 0x7ff1f0145e60 2024-03-29T10:34:52.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1e5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:52.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1e5ffb640 10 client.?.objecter ms_handle_connect 0x7ff1f00af360 2024-03-29T10:34:52.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1e5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:52.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1e5ffb640 10 client.?.objecter ms_handle_connect 0x7ff1f006af60 2024-03-29T10:34:52.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1e5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:52.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1e5ffb640 1 -- 172.21.15.86:0/890627672 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414912296 0 0) 0x7ff1f013b430 con 0x7ff1f0145e60 2024-03-29T10:34:52.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1e5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:34:52.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:52.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:52.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1e5ffb640 10 monclient(hunting): my global_id is 4117 2024-03-29T10:34:52.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1e5ffb640 1 -- 172.21.15.86:0/890627672 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1d4004eb0 con 0x7ff1f0145e60 2024-03-29T10:34:52.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1e5ffb640 1 -- 172.21.15.86:0/890627672 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544632460 0 0) 0x7ff1f01413d0 con 0x7ff1f00af360 2024-03-29T10:34:52.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1e5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:34:52.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:52.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:52.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1e5ffb640 10 monclient(hunting): my global_id is 4125 2024-03-29T10:34:52.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1e5ffb640 1 -- 172.21.15.86:0/890627672 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1f013b430 con 0x7ff1f00af360 2024-03-29T10:34:52.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.244+0000 7ff1e5ffb640 1 -- 172.21.15.86:0/890627672 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176176150 0 0) 0x7ff1f013b5c0 con 0x7ff1f006af60 2024-03-29T10:34:52.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:34:52.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:52.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:52.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 10 monclient(hunting): my global_id is 4121 2024-03-29T10:34:52.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 1 -- 172.21.15.86:0/890627672 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1f01413d0 con 0x7ff1f006af60 2024-03-29T10:34:52.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 1 -- 172.21.15.86:0/890627672 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2003823631 0 0) 0x7ff1d4004eb0 con 0x7ff1f0145e60 2024-03-29T10:34:52.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 1 -- 172.21.15.86:0/890627672 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1f013b5c0 con 0x7ff1f0145e60 2024-03-29T10:34:52.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 1 -- 172.21.15.86:0/890627672 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7ff1e8001c90 con 0x7ff1f0145e60 2024-03-29T10:34:52.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:34:52.244 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 10 monclient(hunting): dump: 2024-03-29T10:34:52.244 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.244 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.244 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.244 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.244 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.244 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.244 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.244 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.245 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.245 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:34:52.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 1 -- 172.21.15.86:0/890627672 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1937824713 0 0) 0x7ff1f013b430 con 0x7ff1f00af360 2024-03-29T10:34:52.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 1 -- 172.21.15.86:0/890627672 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1d4004eb0 con 0x7ff1f00af360 2024-03-29T10:34:52.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 1 -- 172.21.15.86:0/890627672 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7ff1e00018a0 con 0x7ff1f00af360 2024-03-29T10:34:52.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:34:52.245 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 10 monclient(hunting): dump: 2024-03-29T10:34:52.245 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.245 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.245 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.245 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.245 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.246 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.246 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.246 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.246 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.246 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 1 -- 172.21.15.86:0/890627672 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2991973408 0 0) 0x7ff1f01413d0 con 0x7ff1f006af60 2024-03-29T10:34:52.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 1 -- 172.21.15.86:0/890627672 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1f013b430 con 0x7ff1f006af60 2024-03-29T10:34:52.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 1 -- 172.21.15.86:0/890627672 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7ff1ec001b80 con 0x7ff1f006af60 2024-03-29T10:34:52.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:34:52.246 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 10 monclient(hunting): dump: 2024-03-29T10:34:52.246 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.246 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.246 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.247 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.247 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.247 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.247 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.247 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.247 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.247 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 1 -- 172.21.15.86:0/890627672 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2212088754 0 0) 0x7ff1f013b5c0 con 0x7ff1f0145e60 2024-03-29T10:34:52.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 1 -- 172.21.15.86:0/890627672 >> v1:172.21.15.84:6790/0 conn(0x7ff1f006af60 legacy=0x7ff1f013ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 1 -- 172.21.15.86:0/890627672 >> v1:172.21.15.84:6789/0 conn(0x7ff1f00af360 legacy=0x7ff1f01da300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:34:52.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 1 monclient: found mon.b 2024-03-29T10:34:52.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:52.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:52.247 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 1 -- 172.21.15.86:0/890627672 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f013b750 con 0x7ff1f0145e60 2024-03-29T10:34:52.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:34:52.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:34:52.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:52.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1f72ff640 5 monclient: authenticate success, global_id 4117 2024-03-29T10:34:52.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1f72ff640 10 monclient: _renew_subs 2024-03-29T10:34:52.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1f72ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:52.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1f72ff640 1 -- 172.21.15.86:0/890627672 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1f0071950 con 0x7ff1f0145e60 2024-03-29T10:34:52.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 1 -- 172.21.15.86:0/890627672 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1e8003990 con 0x7ff1f0145e60 2024-03-29T10:34:52.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:52.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 1 -- 172.21.15.86:0/890627672 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7ff1e8003d60 con 0x7ff1f0145e60 2024-03-29T10:34:52.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:52.248 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 10 monclient: dump: 2024-03-29T10:34:52.248 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.249 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.249 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.249 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.249 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.249 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.249 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.249 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.249 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.249 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 1 -- 172.21.15.86:0/890627672 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff1e80042d0 con 0x7ff1f0145e60 2024-03-29T10:34:52.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1f72ff640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:34:52.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e77fe640 4 set_mon_vals no callback set 2024-03-29T10:34:52.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1f72ff640 10 monclient: _renew_subs 2024-03-29T10:34:52.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1f72ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:52.249 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1f72ff640 1 -- 172.21.15.86:0/890627672 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1f0140d40 con 0x7ff1f0145e60 2024-03-29T10:34:52.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1f72ff640 1 librados: init done 2024-03-29T10:34:52.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 1 -- 172.21.15.86:0/890627672 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 3189334326 0 0) 0x7ff1e8005ef0 con 0x7ff1f0145e60 2024-03-29T10:34:52.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 10 client.4117.objecter ms_dispatch 0x7ff1f0000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T10:34:52.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 3 client.4117.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T10:34:52.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 3 client.4117.objecter handle_osd_map decoding full epoch 3 2024-03-29T10:34:52.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.245+0000 7ff1e5ffb640 20 client.4117.objecter dump_active .. 0 homeless 2024-03-29T10:34:52.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.246+0000 7ff1f72ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:52.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.246+0000 7ff1f72ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:52.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.246+0000 7ff1f72ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:52.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.246+0000 7ff1f72ff640 1 -- 172.21.15.86:0/890627672 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1f0067c00 con 0x7ff1f0145e60 2024-03-29T10:34:52.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.248+0000 7ff1e5ffb640 1 -- 172.21.15.86:0/890627672 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff1e8006680 con 0x7ff1f0145e60 2024-03-29T10:34:52.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.248+0000 7ff1e5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:52.250 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.248+0000 7ff1e5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:34:52.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.321+0000 7ff1f72ff640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "81f7bc3e-3d25-4774-b43b-8eddffd49bfd", "id": 1}] 2024-03-29T10:34:52.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.321+0000 7ff1f72ff640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "81f7bc3e-3d25-4774-b43b-8eddffd49bfd", "id": 1}] 2024-03-29T10:34:52.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.321+0000 7ff1f72ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:52.306 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.321+0000 7ff1f72ff640 1 -- 172.21.15.86:0/890627672 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd new", "uuid": "81f7bc3e-3d25-4774-b43b-8eddffd49bfd", "id": 1} v 0) v1 -- 0x7ff1f004f730 con 0x7ff1f0145e60 2024-03-29T10:34:52.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.324+0000 7ff1e5ffb640 1 -- 172.21.15.86:0/890627672 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "81f7bc3e-3d25-4774-b43b-8eddffd49bfd", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 1124424202 0 940483276) 0x7ff1e8006820 con 0x7ff1f0145e60 2024-03-29T10:34:52.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.324+0000 7ff1e5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "81f7bc3e-3d25-4774-b43b-8eddffd49bfd", "id": 1}] 2024-03-29T10:34:52.309 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.324+0000 7ff1e5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:34:52.310 INFO:teuthology.orchestra.run.smithi086.stdout:1 2024-03-29T10:34:52.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.325+0000 7ff1f72ff640 10 librados: watch_flush enter 2024-03-29T10:34:52.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.325+0000 7ff1f72ff640 10 librados: watch_flush exit 2024-03-29T10:34:52.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.325+0000 7ff1f72ff640 20 client.4117.objecter shutdown clearing up homeless session... 2024-03-29T10:34:52.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.325+0000 7ff1f72ff640 10 client.4117.objecter successfully canceled tick 2024-03-29T10:34:52.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.325+0000 7ff1f72ff640 10 monclient: shutdown 2024-03-29T10:34:52.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.325+0000 7ff1f72ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:52.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.325+0000 7ff1f72ff640 1 -- 172.21.15.86:0/890627672 >> v1:172.21.15.86:6789/0 conn(0x7ff1f0145e60 legacy=0x7ff1f01daa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.325+0000 7ff1f72ff640 1 -- 172.21.15.86:0/890627672 shutdown_connections 2024-03-29T10:34:52.311 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.325+0000 7ff1f72ff640 1 -- 172.21.15.86:0/890627672 >> 172.21.15.86:0/890627672 conn(0x7ff1f00b58f0 msgr2=0x7ff1f00b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:52.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.325+0000 7ff1f72ff640 1 -- 172.21.15.86:0/890627672 shutdown_connections 2024-03-29T10:34:52.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.325+0000 7ff1f72ff640 1 -- 172.21.15.86:0/890627672 wait complete. 2024-03-29T10:34:52.312 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.325+0000 7ff1f72ff640 1 librados: shutdown 2024-03-29T10:34:52.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 47ef622f-d2a9-47d5-b13e-3618600661eb 2 2024-03-29T10:34:52.417 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-29T10:34:52.430+0000 7f4e58c21040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:34:52.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.435+0000 7fb91c0a8640 10 monclient: get_monmap_and_config 2024-03-29T10:34:52.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.435+0000 7fb91c0a8640 10 monclient: build_initial_monmap 2024-03-29T10:34:52.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.435+0000 7fb91c0a8640 10 monclient: monmap: 2024-03-29T10:34:52.421 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-29T10:34:52.421 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.421 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:52.436965+0000 2024-03-29T10:34:52.421 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:52.436965+0000 2024-03-29T10:34:52.421 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:52.421 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.422 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:34:52.422 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:34:52.422 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:34:52.422 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.436+0000 7fb91c0a8640 1 Processor -- start 2024-03-29T10:34:52.428 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T10:34:52.437+0000 7f187bc42040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:34:52.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.436+0000 7fb91c0a8640 1 -- start start 2024-03-29T10:34:52.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.436+0000 7fb91c0a8640 10 monclient: init 2024-03-29T10:34:52.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb91c0a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:52.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb91c0a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:34:52.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb91c0a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb91409ac40 addr v1:172.21.15.86:6789/0 2024-03-29T10:34:52.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb91c0a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb91409c870 addr v1:172.21.15.84:6790/0 2024-03-29T10:34:52.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb91c0a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb9140b4360 addr v1:172.21.15.84:6789/0 2024-03-29T10:34:52.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb91c0a8640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb91409dec0 con 0x7fb9140b4360 2024-03-29T10:34:52.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb91c0a8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb91409da30 con 0x7fb91409ac40 2024-03-29T10:34:52.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb91c0a8640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9140b39c0 con 0x7fb91409c870 2024-03-29T10:34:52.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb91c0a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:52.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb91c0a8640 10 monclient(hunting): authenticate will time out at 776.602364s 2024-03-29T10:34:52.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb919e44640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fb91409ac40 0x7fb9140b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:50284/0 (socket says 172.21.15.86:50284) 2024-03-29T10:34:52.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb919e44640 1 -- 172.21.15.86:0/3714373807 learned_addr learned my addr 172.21.15.86:0/3714373807 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T10:34:52.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 1 -- 172.21.15.86:0/3714373807 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1468086404 0 0) 0x7fb91409da30 con 0x7fb91409ac40 2024-03-29T10:34:52.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 10 monclient(hunting): _negotiate 2024-03-29T10:34:52.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:52.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:52.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 10 monclient(hunting): my global_id is 4120 2024-03-29T10:34:52.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 1 -- 172.21.15.86:0/3714373807 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb900003520 con 0x7fb91409ac40 2024-03-29T10:34:52.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 1 -- 172.21.15.86:0/3714373807 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2520813463 0 0) 0x7fb90c001490 con 0x7fb9140b4360 2024-03-29T10:34:52.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 10 monclient(hunting): _negotiate 2024-03-29T10:34:52.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:52.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:52.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 10 monclient(hunting): my global_id is 4128 2024-03-29T10:34:52.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 1 -- 172.21.15.86:0/3714373807 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb91409da30 con 0x7fb9140b4360 2024-03-29T10:34:52.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 1 -- 172.21.15.86:0/3714373807 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854062165 0 0) 0x7fb9140b39c0 con 0x7fb91409c870 2024-03-29T10:34:52.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 10 monclient(hunting): _negotiate 2024-03-29T10:34:52.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:52.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:52.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 10 monclient(hunting): my global_id is 4124 2024-03-29T10:34:52.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 1 -- 172.21.15.86:0/3714373807 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb90c001490 con 0x7fb91409c870 2024-03-29T10:34:52.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 1 -- 172.21.15.86:0/3714373807 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2947668095 0 0) 0x7fb900003520 con 0x7fb91409ac40 2024-03-29T10:34:52.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 1 -- 172.21.15.86:0/3714373807 >> v1:172.21.15.84:6790/0 conn(0x7fb91409c870 legacy=0x7fb91413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 1 -- 172.21.15.86:0/3714373807 >> v1:172.21.15.84:6789/0 conn(0x7fb9140b4360 legacy=0x7fb91414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 10 monclient: _finish_hunting 0 2024-03-29T10:34:52.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 1 monclient: found mon.noname-c 2024-03-29T10:34:52.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:52.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-29T10:34:52.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 1 -- 172.21.15.86:0/3714373807 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9140b5160 con 0x7fb91409ac40 2024-03-29T10:34:52.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 10 monclient: _finish_auth 0 2024-03-29T10:34:52.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 10 monclient: _check_auth_tickets 2024-03-29T10:34:52.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:52.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb91c0a8640 5 monclient: authenticate success, global_id 4120 2024-03-29T10:34:52.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 1 -- 172.21.15.86:0/3714373807 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fb910001690 con 0x7fb91409ac40 2024-03-29T10:34:52.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb91c0a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:52.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:52.435 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 10 monclient: dump: 2024-03-29T10:34:52.435 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.435 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.435 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.435 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.435 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.435 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.435 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.435 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.435 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.436 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb91c0a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:52.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 1 -- 172.21.15.86:0/3714373807 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb910001870 con 0x7fb91409ac40 2024-03-29T10:34:52.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:52.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 1 -- 172.21.15.86:0/3714373807 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fb910001c80 con 0x7fb91409ac40 2024-03-29T10:34:52.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb91c0a8640 10 monclient: get_monmap_and_config success 2024-03-29T10:34:52.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb91c0a8640 4 set_mon_vals no callback set 2024-03-29T10:34:52.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:52.436 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.437+0000 7fb918e42640 10 monclient: dump: 2024-03-29T10:34:52.436 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.436 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.436 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.436 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.437 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.437 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.437 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.437 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.437 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.437 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.438+0000 7fb91c0a8640 10 monclient: shutdown 2024-03-29T10:34:52.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.438+0000 7fb91c0a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:52.437 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.438+0000 7fb91c0a8640 1 -- 172.21.15.86:0/3714373807 >> v1:172.21.15.86:6789/0 conn(0x7fb91409ac40 legacy=0x7fb9140b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.438+0000 7fb91c0a8640 1 -- 172.21.15.86:0/3714373807 shutdown_connections 2024-03-29T10:34:52.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.438+0000 7fb91c0a8640 1 -- 172.21.15.86:0/3714373807 >> 172.21.15.86:0/3714373807 conn(0x7fb914066b00 msgr2=0x7fb914066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:52.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.438+0000 7fb91c0a8640 1 -- 172.21.15.86:0/3714373807 shutdown_connections 2024-03-29T10:34:52.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.438+0000 7fb91c0a8640 1 -- 172.21.15.86:0/3714373807 wait complete. 2024-03-29T10:34:52.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.439+0000 7fb91c0a8640 10 monclient: build_initial_monmap 2024-03-29T10:34:52.438 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.439+0000 7fb91c0a8640 10 monclient: monmap: 2024-03-29T10:34:52.438 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-29T10:34:52.438 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:34:52.438 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-03-29T10:34:52.438 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-03-29T10:34:52.438 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:52.438 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.438 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:34:52.439 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:34:52.439 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:34:52.439 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.439+0000 7fb91c0a8640 1 librados: starting msgr at 2024-03-29T10:34:52.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.439+0000 7fb91c0a8640 1 librados: starting objecter 2024-03-29T10:34:52.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.439+0000 7fb91c0a8640 1 Processor -- start 2024-03-29T10:34:52.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.439+0000 7fb91c0a8640 1 -- start start 2024-03-29T10:34:52.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.439+0000 7fb91c0a8640 1 librados: setting wanted keys 2024-03-29T10:34:52.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.439+0000 7fb91c0a8640 1 librados: calling monclient init 2024-03-29T10:34:52.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.439+0000 7fb91c0a8640 10 monclient: init 2024-03-29T10:34:52.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb91c0a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:52.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb91c0a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:34:52.439 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb91c0a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb91409ac40 addr v1:172.21.15.86:6789/0 2024-03-29T10:34:52.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb91c0a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb9140b1860 addr v1:172.21.15.84:6789/0 2024-03-29T10:34:52.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb91c0a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb91413b5b0 addr v1:172.21.15.84:6790/0 2024-03-29T10:34:52.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb91c0a8640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9141d7660 con 0x7fb9140b1860 2024-03-29T10:34:52.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb91c0a8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9140ade20 con 0x7fb91409ac40 2024-03-29T10:34:52.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb91c0a8640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9141406b0 con 0x7fb91413b5b0 2024-03-29T10:34:52.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb91c0a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:52.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb91c0a8640 10 monclient(hunting): authenticate will time out at 776.605282s 2024-03-29T10:34:52.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb919e44640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fb91409ac40 0x7fb91413ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:50294/0 (socket says 172.21.15.86:50294) 2024-03-29T10:34:52.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb919e44640 1 -- 172.21.15.86:0/3987764035 learned_addr learned my addr 172.21.15.86:0/3987764035 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T10:34:52.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 client.?.objecter ms_handle_connect 0x7fb91409ac40 2024-03-29T10:34:52.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:52.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 client.?.objecter ms_handle_connect 0x7fb9140b1860 2024-03-29T10:34:52.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:52.440 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 1 -- 172.21.15.86:0/3987764035 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1114987771 0 0) 0x7fb9140ade20 con 0x7fb91409ac40 2024-03-29T10:34:52.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:34:52.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:52.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:52.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 monclient(hunting): my global_id is 4123 2024-03-29T10:34:52.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 1 -- 172.21.15.86:0/3987764035 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8f8004eb0 con 0x7fb91409ac40 2024-03-29T10:34:52.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 client.?.objecter ms_handle_connect 0x7fb91413b5b0 2024-03-29T10:34:52.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:52.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 1 -- 172.21.15.86:0/3987764035 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 590808001 0 0) 0x7fb9141d7660 con 0x7fb9140b1860 2024-03-29T10:34:52.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:34:52.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:52.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:52.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 monclient(hunting): my global_id is 4131 2024-03-29T10:34:52.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 1 -- 172.21.15.86:0/3987764035 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9140ade20 con 0x7fb9140b1860 2024-03-29T10:34:52.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 1 -- 172.21.15.86:0/3987764035 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2237416995 0 0) 0x7fb9141406b0 con 0x7fb91413b5b0 2024-03-29T10:34:52.441 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:34:52.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:52.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:52.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 monclient(hunting): my global_id is 4127 2024-03-29T10:34:52.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 1 -- 172.21.15.86:0/3987764035 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9141d7660 con 0x7fb91413b5b0 2024-03-29T10:34:52.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 1 -- 172.21.15.86:0/3987764035 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2872921149 0 0) 0x7fb8f8004eb0 con 0x7fb91409ac40 2024-03-29T10:34:52.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 1 -- 172.21.15.86:0/3987764035 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9141406b0 con 0x7fb91409ac40 2024-03-29T10:34:52.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 1 -- 172.21.15.86:0/3987764035 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fb910001bc0 con 0x7fb91409ac40 2024-03-29T10:34:52.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:34:52.442 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 monclient(hunting): dump: 2024-03-29T10:34:52.443 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.443 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.443 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.443 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.443 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.443 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.443 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.443 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.443 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.443 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:34:52.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 1 -- 172.21.15.86:0/3987764035 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2834539135 0 0) 0x7fb9140ade20 con 0x7fb9140b1860 2024-03-29T10:34:52.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 1 -- 172.21.15.86:0/3987764035 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb8f8004eb0 con 0x7fb9140b1860 2024-03-29T10:34:52.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 1 -- 172.21.15.86:0/3987764035 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fb904001870 con 0x7fb9140b1860 2024-03-29T10:34:52.443 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:34:52.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 monclient(hunting): dump: 2024-03-29T10:34:52.444 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.444 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.444 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.444 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.444 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.444 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.444 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.444 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.444 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.444 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 1 -- 172.21.15.86:0/3987764035 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1262885102 0 0) 0x7fb9141d7660 con 0x7fb91413b5b0 2024-03-29T10:34:52.444 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 1 -- 172.21.15.86:0/3987764035 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9140ade20 con 0x7fb91413b5b0 2024-03-29T10:34:52.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 1 -- 172.21.15.86:0/3987764035 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fb90c001a00 con 0x7fb91413b5b0 2024-03-29T10:34:52.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:34:52.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 monclient(hunting): dump: 2024-03-29T10:34:52.445 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.445 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.445 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.445 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.445 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.445 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.445 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.445 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.445 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.445 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.445 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 1 -- 172.21.15.86:0/3987764035 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2982849853 0 0) 0x7fb9141406b0 con 0x7fb91409ac40 2024-03-29T10:34:52.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 1 -- 172.21.15.86:0/3987764035 >> v1:172.21.15.84:6790/0 conn(0x7fb91413b5b0 legacy=0x7fb9141e2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 1 -- 172.21.15.86:0/3987764035 >> v1:172.21.15.84:6789/0 conn(0x7fb9140b1860 legacy=0x7fb91413aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:34:52.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 1 monclient: found mon.b 2024-03-29T10:34:52.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:52.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:52.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 1 -- 172.21.15.86:0/3987764035 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9141cccd0 con 0x7fb91409ac40 2024-03-29T10:34:52.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:34:52.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:34:52.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb90a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:52.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb91c0a8640 5 monclient: authenticate success, global_id 4123 2024-03-29T10:34:52.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb91c0a8640 10 monclient: _renew_subs 2024-03-29T10:34:52.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb91c0a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:52.446 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.440+0000 7fb91c0a8640 1 -- 172.21.15.86:0/3987764035 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9141cce60 con 0x7fb91409ac40 2024-03-29T10:34:52.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.441+0000 7fb91c0a8640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:34:52.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.441+0000 7fb91c0a8640 10 monclient: _renew_subs 2024-03-29T10:34:52.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.441+0000 7fb91c0a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:52.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.441+0000 7fb91c0a8640 1 -- 172.21.15.86:0/3987764035 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9141d7190 con 0x7fb91409ac40 2024-03-29T10:34:52.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.441+0000 7fb91c0a8640 1 librados: init done 2024-03-29T10:34:52.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.441+0000 7fb90a7fc640 1 -- 172.21.15.86:0/3987764035 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb910004bf0 con 0x7fb91409ac40 2024-03-29T10:34:52.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.441+0000 7fb90a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:52.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.441+0000 7fb90a7fc640 1 -- 172.21.15.86:0/3987764035 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fb910004f00 con 0x7fb91409ac40 2024-03-29T10:34:52.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.441+0000 7fb90a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.441+0000 7fb90a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:52.447 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.441+0000 7fb90a7fc640 10 monclient: dump: 2024-03-29T10:34:52.447 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.447 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.447 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.447 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.448 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.448 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.448 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.448 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.448 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.448 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.441+0000 7fb90bfff640 4 set_mon_vals no callback set 2024-03-29T10:34:52.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.441+0000 7fb90a7fc640 1 -- 172.21.15.86:0/3987764035 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb9100053f0 con 0x7fb91409ac40 2024-03-29T10:34:52.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.441+0000 7fb90a7fc640 1 -- 172.21.15.86:0/3987764035 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 1668316765 0 0) 0x7fb910005ac0 con 0x7fb91409ac40 2024-03-29T10:34:52.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.441+0000 7fb90a7fc640 10 client.4123.objecter ms_dispatch 0x7fb914000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T10:34:52.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.441+0000 7fb90a7fc640 3 client.4123.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T10:34:52.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.441+0000 7fb90a7fc640 3 client.4123.objecter handle_osd_map decoding full epoch 4 2024-03-29T10:34:52.448 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.441+0000 7fb90a7fc640 20 client.4123.objecter dump_active .. 0 homeless 2024-03-29T10:34:52.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.441+0000 7fb91c0a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:52.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.441+0000 7fb91c0a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:52.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.441+0000 7fb91c0a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:52.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.441+0000 7fb91c0a8640 1 -- 172.21.15.86:0/3987764035 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8fc005080 con 0x7fb91409ac40 2024-03-29T10:34:52.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.443+0000 7fb90a7fc640 1 -- 172.21.15.86:0/3987764035 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb910005e20 con 0x7fb91409ac40 2024-03-29T10:34:52.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.443+0000 7fb90a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:52.449 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.443+0000 7fb90a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:34:52.472 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-29T10:34:52.487+0000 7f4e58c21040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:34:52.477 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T10:34:52.491+0000 7f187bc42040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:34:52.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.517+0000 7fb91c0a8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "47ef622f-d2a9-47d5-b13e-3618600661eb", "id": 2}] 2024-03-29T10:34:52.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.517+0000 7fb91c0a8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "47ef622f-d2a9-47d5-b13e-3618600661eb", "id": 2}] 2024-03-29T10:34:52.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.517+0000 7fb91c0a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:52.503 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.517+0000 7fb91c0a8640 1 -- 172.21.15.86:0/3987764035 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd new", "uuid": "47ef622f-d2a9-47d5-b13e-3618600661eb", "id": 2} v 0) v1 -- 0x7fb8fc0058b0 con 0x7fb91409ac40 2024-03-29T10:34:52.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.521+0000 7fb90a7fc640 1 -- 172.21.15.86:0/3987764035 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "47ef622f-d2a9-47d5-b13e-3618600661eb", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 4022955273 0 216599125) 0x7fb9100056a0 con 0x7fb91409ac40 2024-03-29T10:34:52.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.521+0000 7fb90a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "47ef622f-d2a9-47d5-b13e-3618600661eb", "id": 2}] 2024-03-29T10:34:52.506 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.521+0000 7fb90a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:34:52.506 INFO:teuthology.orchestra.run.smithi086.stdout:2 2024-03-29T10:34:52.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.521+0000 7fb91c0a8640 10 librados: watch_flush enter 2024-03-29T10:34:52.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.521+0000 7fb91c0a8640 10 librados: watch_flush exit 2024-03-29T10:34:52.507 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.521+0000 7fb91c0a8640 20 client.4123.objecter shutdown clearing up homeless session... 2024-03-29T10:34:52.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.521+0000 7fb91c0a8640 10 client.4123.objecter successfully canceled tick 2024-03-29T10:34:52.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.521+0000 7fb91c0a8640 10 monclient: shutdown 2024-03-29T10:34:52.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.522+0000 7fb91c0a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:52.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.522+0000 7fb91c0a8640 1 -- 172.21.15.86:0/3987764035 >> v1:172.21.15.86:6789/0 conn(0x7fb91409ac40 legacy=0x7fb91413ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.522+0000 7fb91c0a8640 1 -- 172.21.15.86:0/3987764035 shutdown_connections 2024-03-29T10:34:52.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.522+0000 7fb91c0a8640 1 -- 172.21.15.86:0/3987764035 >> 172.21.15.86:0/3987764035 conn(0x7fb914066b00 msgr2=0x7fb91413cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:52.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.522+0000 7fb91c0a8640 1 -- 172.21.15.86:0/3987764035 shutdown_connections 2024-03-29T10:34:52.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.522+0000 7fb91c0a8640 1 -- 172.21.15.86:0/3987764035 wait complete. 2024-03-29T10:34:52.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.522+0000 7fb91c0a8640 1 librados: shutdown 2024-03-29T10:34:52.517 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new a16650aa-a2da-44e2-bad9-b872009b9413 3 2024-03-29T10:34:52.520 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-29T10:34:52.535+0000 7f4e58c21040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:34:52.523 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T10:34:52.537+0000 7f187bc42040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:34:52.577 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-29T10:34:52.591+0000 7f4e58c21040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:34:52.577 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T10:34:52.591+0000 7f187bc42040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:34:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.631+0000 7f3c183d4640 10 monclient: get_monmap_and_config 2024-03-29T10:34:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.631+0000 7f3c183d4640 10 monclient: build_initial_monmap 2024-03-29T10:34:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.631+0000 7f3c183d4640 10 monclient: monmap: 2024-03-29T10:34:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-29T10:34:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:52.632815+0000 2024-03-29T10:34:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:52.632815+0000 2024-03-29T10:34:52.617 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:34:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:34:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:34:52.618 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.632+0000 7f3c183d4640 1 Processor -- start 2024-03-29T10:34:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.632+0000 7f3c183d4640 1 -- start start 2024-03-29T10:34:52.618 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.632+0000 7f3c183d4640 10 monclient: init 2024-03-29T10:34:52.625 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T10:34:52.633+0000 7f187bc42040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:34:52.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c183d4640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:52.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c183d4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:34:52.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c183d4640 10 monclient(hunting): picked mon.noname-b con 0x7f3c1009ac40 addr v1:172.21.15.84:6790/0 2024-03-29T10:34:52.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c183d4640 10 monclient(hunting): picked mon.noname-a con 0x7f3c1009c870 addr v1:172.21.15.84:6789/0 2024-03-29T10:34:52.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c183d4640 10 monclient(hunting): picked mon.noname-c con 0x7f3c100b4360 addr v1:172.21.15.86:6789/0 2024-03-29T10:34:52.630 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c183d4640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c1009dec0 con 0x7f3c1009c870 2024-03-29T10:34:52.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c183d4640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c1009da30 con 0x7f3c100b4360 2024-03-29T10:34:52.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c183d4640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c100b39c0 con 0x7f3c1009ac40 2024-03-29T10:34:52.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c183d4640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:52.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c183d4640 10 monclient(hunting): authenticate will time out at 776.798245s 2024-03-29T10:34:52.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c16971640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f3c100b4360 0x7f3c1014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:50300/0 (socket says 172.21.15.86:50300) 2024-03-29T10:34:52.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c16971640 1 -- 172.21.15.86:0/1691643741 learned_addr learned my addr 172.21.15.86:0/1691643741 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T10:34:52.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 1 -- 172.21.15.86:0/1691643741 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2922882674 0 0) 0x7f3c1009da30 con 0x7f3c100b4360 2024-03-29T10:34:52.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 10 monclient(hunting): _negotiate 2024-03-29T10:34:52.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:52.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:52.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 10 monclient(hunting): my global_id is 4126 2024-03-29T10:34:52.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 1 -- 172.21.15.86:0/1691643741 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bfc003520 con 0x7f3c100b4360 2024-03-29T10:34:52.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 1 -- 172.21.15.86:0/1691643741 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204746964 0 0) 0x7f3c100b39c0 con 0x7f3c1009ac40 2024-03-29T10:34:52.631 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 10 monclient(hunting): _negotiate 2024-03-29T10:34:52.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:52.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:52.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 10 monclient(hunting): my global_id is 4130 2024-03-29T10:34:52.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 1 -- 172.21.15.86:0/1691643741 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c1009da30 con 0x7f3c1009ac40 2024-03-29T10:34:52.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 1 -- 172.21.15.86:0/1691643741 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777659771 0 0) 0x7f3c0c001110 con 0x7f3c1009c870 2024-03-29T10:34:52.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 10 monclient(hunting): _negotiate 2024-03-29T10:34:52.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:52.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:52.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 10 monclient(hunting): my global_id is 4134 2024-03-29T10:34:52.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 1 -- 172.21.15.86:0/1691643741 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c100b39c0 con 0x7f3c1009c870 2024-03-29T10:34:52.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 1 -- 172.21.15.86:0/1691643741 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4044392568 0 0) 0x7f3bfc003520 con 0x7f3c100b4360 2024-03-29T10:34:52.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 1 -- 172.21.15.86:0/1691643741 >> v1:172.21.15.84:6790/0 conn(0x7f3c1009ac40 legacy=0x7f3c100b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 1 -- 172.21.15.86:0/1691643741 >> v1:172.21.15.84:6789/0 conn(0x7f3c1009c870 legacy=0x7f3c1013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 10 monclient: _finish_hunting 0 2024-03-29T10:34:52.632 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 1 monclient: found mon.noname-c 2024-03-29T10:34:52.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:52.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-29T10:34:52.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 1 -- 172.21.15.86:0/1691643741 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c100b5160 con 0x7f3c100b4360 2024-03-29T10:34:52.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 10 monclient: _finish_auth 0 2024-03-29T10:34:52.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 10 monclient: _check_auth_tickets 2024-03-29T10:34:52.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:52.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c183d4640 5 monclient: authenticate success, global_id 4126 2024-03-29T10:34:52.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 1 -- 172.21.15.86:0/1691643741 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f3c08001a10 con 0x7f3c100b4360 2024-03-29T10:34:52.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c183d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:52.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:52.633 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 10 monclient: dump: 2024-03-29T10:34:52.633 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.633 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.633 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.634 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.634 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.634 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.634 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.634 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.634 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.634 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c183d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:52.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 1 -- 172.21.15.86:0/1691643741 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c08001bf0 con 0x7f3c100b4360 2024-03-29T10:34:52.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:52.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c183d4640 10 monclient: get_monmap_and_config success 2024-03-29T10:34:52.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 1 -- 172.21.15.86:0/1691643741 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f3c08002000 con 0x7f3c100b4360 2024-03-29T10:34:52.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.634 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c183d4640 4 set_mon_vals no callback set 2024-03-29T10:34:52.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:52.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.633+0000 7f3c1516e640 10 monclient: dump: 2024-03-29T10:34:52.635 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.635 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.635 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.635 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.635 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.635 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.635 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.635 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.635 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.635 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.634+0000 7f3c183d4640 10 monclient: shutdown 2024-03-29T10:34:52.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.634+0000 7f3c183d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:52.635 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.634+0000 7f3c183d4640 1 -- 172.21.15.86:0/1691643741 >> v1:172.21.15.86:6789/0 conn(0x7f3c100b4360 legacy=0x7f3c1014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.634+0000 7f3c183d4640 1 -- 172.21.15.86:0/1691643741 shutdown_connections 2024-03-29T10:34:52.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.634+0000 7f3c183d4640 1 -- 172.21.15.86:0/1691643741 >> 172.21.15.86:0/1691643741 conn(0x7f3c10066b00 msgr2=0x7f3c10066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:52.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.634+0000 7f3c183d4640 1 -- 172.21.15.86:0/1691643741 shutdown_connections 2024-03-29T10:34:52.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.634+0000 7f3c183d4640 1 -- 172.21.15.86:0/1691643741 wait complete. 2024-03-29T10:34:52.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.635+0000 7f3c183d4640 10 monclient: build_initial_monmap 2024-03-29T10:34:52.636 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.635+0000 7f3c183d4640 10 monclient: monmap: 2024-03-29T10:34:52.636 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-29T10:34:52.636 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:34:52.636 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-03-29T10:34:52.636 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-03-29T10:34:52.636 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:52.636 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.636 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:34:52.636 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:34:52.637 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:34:52.637 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.635+0000 7f3c183d4640 1 librados: starting msgr at 2024-03-29T10:34:52.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.635+0000 7f3c183d4640 1 librados: starting objecter 2024-03-29T10:34:52.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.635+0000 7f3c183d4640 1 Processor -- start 2024-03-29T10:34:52.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.635+0000 7f3c183d4640 1 -- start start 2024-03-29T10:34:52.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.635+0000 7f3c183d4640 1 librados: setting wanted keys 2024-03-29T10:34:52.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.635+0000 7f3c183d4640 1 librados: calling monclient init 2024-03-29T10:34:52.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.635+0000 7f3c183d4640 10 monclient: init 2024-03-29T10:34:52.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c183d4640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:52.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c183d4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:34:52.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c183d4640 10 monclient(hunting): picked mon.noname-c con 0x7f3c1009ac40 addr v1:172.21.15.84:6790/0 2024-03-29T10:34:52.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c183d4640 10 monclient(hunting): picked mon.noname-b con 0x7f3c100b7310 addr v1:172.21.15.86:6789/0 2024-03-29T10:34:52.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c183d4640 10 monclient(hunting): picked mon.noname-a con 0x7f3c101d7de0 addr v1:172.21.15.84:6789/0 2024-03-29T10:34:52.637 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c183d4640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c100b78c0 con 0x7f3c101d7de0 2024-03-29T10:34:52.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c183d4640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c100b7d50 con 0x7f3c100b7310 2024-03-29T10:34:52.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c183d4640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c10146480 con 0x7f3c1009ac40 2024-03-29T10:34:52.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c183d4640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:52.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c183d4640 10 monclient(hunting): authenticate will time out at 776.801272s 2024-03-29T10:34:52.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c1596f640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f3c100b7310 0x7f3c101d76d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:50314/0 (socket says 172.21.15.86:50314) 2024-03-29T10:34:52.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c1596f640 1 -- 172.21.15.86:0/1483267763 learned_addr learned my addr 172.21.15.86:0/1483267763 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T10:34:52.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 client.?.objecter ms_handle_connect 0x7f3c100b7310 2024-03-29T10:34:52.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:52.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 client.?.objecter ms_handle_connect 0x7f3c1009ac40 2024-03-29T10:34:52.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:52.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 client.?.objecter ms_handle_connect 0x7f3c101d7de0 2024-03-29T10:34:52.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:52.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 1 -- 172.21.15.86:0/1483267763 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2279536399 0 0) 0x7f3c100b7d50 con 0x7f3c100b7310 2024-03-29T10:34:52.638 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:52.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:52.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:52.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 monclient(hunting): my global_id is 4129 2024-03-29T10:34:52.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 1 -- 172.21.15.86:0/1483267763 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bf4004eb0 con 0x7f3c100b7310 2024-03-29T10:34:52.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 1 -- 172.21.15.86:0/1483267763 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2755559550 0 0) 0x7f3c10146480 con 0x7f3c1009ac40 2024-03-29T10:34:52.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:52.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:52.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:52.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 monclient(hunting): my global_id is 4133 2024-03-29T10:34:52.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 1 -- 172.21.15.86:0/1483267763 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c100b7d50 con 0x7f3c1009ac40 2024-03-29T10:34:52.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 1 -- 172.21.15.86:0/1483267763 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082938696 0 0) 0x7f3c100b78c0 con 0x7f3c101d7de0 2024-03-29T10:34:52.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:52.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:52.639 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:52.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 monclient(hunting): my global_id is 4137 2024-03-29T10:34:52.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 1 -- 172.21.15.86:0/1483267763 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c10146480 con 0x7f3c101d7de0 2024-03-29T10:34:52.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 1 -- 172.21.15.86:0/1483267763 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3965140884 0 0) 0x7f3bf4004eb0 con 0x7f3c100b7310 2024-03-29T10:34:52.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 1 -- 172.21.15.86:0/1483267763 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c100b78c0 con 0x7f3c100b7310 2024-03-29T10:34:52.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 1 -- 172.21.15.86:0/1483267763 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f3c0c0017e0 con 0x7f3c100b7310 2024-03-29T10:34:52.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:34:52.640 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 monclient(hunting): dump: 2024-03-29T10:34:52.640 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.640 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.640 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.640 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.640 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.640 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.640 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.641 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.641 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.641 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:34:52.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 1 -- 172.21.15.86:0/1483267763 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3994783419 0 0) 0x7f3c100b7d50 con 0x7f3c1009ac40 2024-03-29T10:34:52.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 1 -- 172.21.15.86:0/1483267763 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3bf4004eb0 con 0x7f3c1009ac40 2024-03-29T10:34:52.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 1 -- 172.21.15.86:0/1483267763 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 884878343 0 0) 0x7f3c10146480 con 0x7f3c101d7de0 2024-03-29T10:34:52.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 1 -- 172.21.15.86:0/1483267763 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c100b7d50 con 0x7f3c101d7de0 2024-03-29T10:34:52.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 1 -- 172.21.15.86:0/1483267763 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f3c00001920 con 0x7f3c1009ac40 2024-03-29T10:34:52.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:34:52.641 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 monclient(hunting): dump: 2024-03-29T10:34:52.641 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.641 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.641 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.642 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.642 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.642 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.642 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.642 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.642 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.642 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 1 -- 172.21.15.86:0/1483267763 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f3c08001d60 con 0x7f3c101d7de0 2024-03-29T10:34:52.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:34:52.642 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 monclient(hunting): dump: 2024-03-29T10:34:52.642 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.642 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.643 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.643 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.643 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.643 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.643 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.643 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.643 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.643 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 1 -- 172.21.15.86:0/1483267763 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2418694766 0 0) 0x7f3c100b78c0 con 0x7f3c100b7310 2024-03-29T10:34:52.643 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 1 -- 172.21.15.86:0/1483267763 >> v1:172.21.15.84:6790/0 conn(0x7f3c1009ac40 legacy=0x7f3c100b6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 1 -- 172.21.15.86:0/1483267763 >> v1:172.21.15.84:6789/0 conn(0x7f3c101d7de0 legacy=0x7f3c101e2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:34:52.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 1 monclient: found mon.b 2024-03-29T10:34:52.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:52.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:52.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 1 -- 172.21.15.86:0/1483267763 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c101e68d0 con 0x7f3c100b7310 2024-03-29T10:34:52.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 monclient: _finish_auth 0 2024-03-29T10:34:52.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:34:52.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:52.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c183d4640 5 monclient: authenticate success, global_id 4129 2024-03-29T10:34:52.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c183d4640 10 monclient: _renew_subs 2024-03-29T10:34:52.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c183d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:52.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.636+0000 7f3c183d4640 1 -- 172.21.15.86:0/1483267763 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c101e38a0 con 0x7f3c100b7310 2024-03-29T10:34:52.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.637+0000 7f3c183d4640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:34:52.644 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.637+0000 7f3c183d4640 10 monclient: _renew_subs 2024-03-29T10:34:52.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.637+0000 7f3c183d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:52.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.637+0000 7f3c183d4640 1 -- 172.21.15.86:0/1483267763 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c101e3dd0 con 0x7f3c100b7310 2024-03-29T10:34:52.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.637+0000 7f3c183d4640 1 librados: init done 2024-03-29T10:34:52.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.637+0000 7f3c06ffd640 1 -- 172.21.15.86:0/1483267763 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c0c001d50 con 0x7f3c100b7310 2024-03-29T10:34:52.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.637+0000 7f3c06ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:52.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.637+0000 7f3c06ffd640 1 -- 172.21.15.86:0/1483267763 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f3c0c0037b0 con 0x7f3c100b7310 2024-03-29T10:34:52.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.637+0000 7f3c06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.637+0000 7f3c1496d640 4 set_mon_vals no callback set 2024-03-29T10:34:52.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.637+0000 7f3c06ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:52.645 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.637+0000 7f3c06ffd640 10 monclient: dump: 2024-03-29T10:34:52.645 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.645 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.645 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.645 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.646 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.646 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.646 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.646 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.646 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.646 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.637+0000 7f3c06ffd640 1 -- 172.21.15.86:0/1483267763 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3c0c003ca0 con 0x7f3c100b7310 2024-03-29T10:34:52.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.637+0000 7f3c06ffd640 1 -- 172.21.15.86:0/1483267763 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 4216167075 0 0) 0x7f3c0c003f50 con 0x7f3c100b7310 2024-03-29T10:34:52.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.637+0000 7f3c06ffd640 10 client.4129.objecter ms_dispatch 0x7f3c10000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T10:34:52.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.637+0000 7f3c06ffd640 3 client.4129.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T10:34:52.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.637+0000 7f3c06ffd640 3 client.4129.objecter handle_osd_map decoding full epoch 5 2024-03-29T10:34:52.646 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.637+0000 7f3c06ffd640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-29T10:34:52.647 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-29T10:34:52.637+0000 7f4e58c21040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:34:52.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.637+0000 7f3c183d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:52.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.637+0000 7f3c183d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:52.647 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.637+0000 7f3c183d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:52.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.637+0000 7f3c183d4640 1 -- 172.21.15.86:0/1483267763 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bf8005080 con 0x7f3c100b7310 2024-03-29T10:34:52.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.639+0000 7f3c06ffd640 1 -- 172.21.15.86:0/1483267763 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f3c0c0057c0 con 0x7f3c100b7310 2024-03-29T10:34:52.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.639+0000 7f3c06ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:52.648 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.639+0000 7f3c06ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:34:52.697 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.712+0000 7f3c183d4640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a16650aa-a2da-44e2-bad9-b872009b9413", "id": 3}] 2024-03-29T10:34:52.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.712+0000 7f3c183d4640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a16650aa-a2da-44e2-bad9-b872009b9413", "id": 3}] 2024-03-29T10:34:52.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.712+0000 7f3c183d4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:52.698 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.712+0000 7f3c183d4640 1 -- 172.21.15.86:0/1483267763 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd new", "uuid": "a16650aa-a2da-44e2-bad9-b872009b9413", "id": 3} v 0) v1 -- 0x7f3bf80058b0 con 0x7f3c100b7310 2024-03-29T10:34:52.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.716+0000 7f3c06ffd640 1 -- 172.21.15.86:0/1483267763 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a16650aa-a2da-44e2-bad9-b872009b9413", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 2775830424 0 525046306) 0x7f3c0c005ab0 con 0x7f3c100b7310 2024-03-29T10:34:52.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.716+0000 7f3c06ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a16650aa-a2da-44e2-bad9-b872009b9413", "id": 3}] 2024-03-29T10:34:52.701 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.716+0000 7f3c06ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:34:52.702 INFO:teuthology.orchestra.run.smithi086.stdout:3 2024-03-29T10:34:52.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.717+0000 7f3c183d4640 10 librados: watch_flush enter 2024-03-29T10:34:52.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.717+0000 7f3c183d4640 10 librados: watch_flush exit 2024-03-29T10:34:52.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.717+0000 7f3c183d4640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-29T10:34:52.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.717+0000 7f3c183d4640 10 client.4129.objecter successfully canceled tick 2024-03-29T10:34:52.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.717+0000 7f3c183d4640 10 monclient: shutdown 2024-03-29T10:34:52.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.717+0000 7f3c183d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:52.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.717+0000 7f3c183d4640 1 -- 172.21.15.86:0/1483267763 >> v1:172.21.15.86:6789/0 conn(0x7f3c100b7310 legacy=0x7f3c101d76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.717+0000 7f3c183d4640 1 -- 172.21.15.86:0/1483267763 shutdown_connections 2024-03-29T10:34:52.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.717+0000 7f3c183d4640 1 -- 172.21.15.86:0/1483267763 >> 172.21.15.86:0/1483267763 conn(0x7f3c10066b00 msgr2=0x7f3c1013a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:52.703 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.717+0000 7f3c183d4640 1 -- 172.21.15.86:0/1483267763 shutdown_connections 2024-03-29T10:34:52.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.717+0000 7f3c183d4640 1 -- 172.21.15.86:0/1483267763 wait complete. 2024-03-29T10:34:52.704 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.717+0000 7f3c183d4640 1 librados: shutdown 2024-03-29T10:34:52.712 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 0c8cbbef-4bb9-475f-b150-0e97365d0d56 4 2024-03-29T10:34:52.736 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T10:34:52.750+0000 7f187bc42040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:34:52.743 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-29T10:34:52.758+0000 7f4e58c21040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:34:52.806 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T10:34:52.819+0000 7f187bc42040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:34:52.811 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.826+0000 7fa89f9d6640 10 monclient: get_monmap_and_config 2024-03-29T10:34:52.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.826+0000 7fa89f9d6640 10 monclient: build_initial_monmap 2024-03-29T10:34:52.812 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.826+0000 7fa89f9d6640 10 monclient: monmap: 2024-03-29T10:34:52.812 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-29T10:34:52.812 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.812 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:52.827847+0000 2024-03-29T10:34:52.812 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:52.827847+0000 2024-03-29T10:34:52.812 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:52.812 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.812 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:34:52.812 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:34:52.812 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:34:52.812 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.827+0000 7fa89f9d6640 1 Processor -- start 2024-03-29T10:34:52.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.827+0000 7fa89f9d6640 1 -- start start 2024-03-29T10:34:52.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.827+0000 7fa89f9d6640 10 monclient: init 2024-03-29T10:34:52.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa89f9d6640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:52.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa89f9d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:34:52.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa89f9d6640 10 monclient(hunting): picked mon.noname-a con 0x7fa89809ac40 addr v1:172.21.15.84:6789/0 2024-03-29T10:34:52.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa89f9d6640 10 monclient(hunting): picked mon.noname-c con 0x7fa89809c870 addr v1:172.21.15.86:6789/0 2024-03-29T10:34:52.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa89f9d6640 10 monclient(hunting): picked mon.noname-b con 0x7fa8980b4360 addr v1:172.21.15.84:6790/0 2024-03-29T10:34:52.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa89f9d6640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa89809dec0 con 0x7fa89809ac40 2024-03-29T10:34:52.822 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa89f9d6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa89809da30 con 0x7fa89809c870 2024-03-29T10:34:52.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa89f9d6640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8980b39c0 con 0x7fa8980b4360 2024-03-29T10:34:52.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa89f9d6640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:52.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa89f9d6640 10 monclient(hunting): authenticate will time out at 776.993382s 2024-03-29T10:34:52.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa89cf71640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa89809c870 0x7fa89813f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:50322/0 (socket says 172.21.15.86:50322) 2024-03-29T10:34:52.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa89cf71640 1 -- 172.21.15.86:0/406479810 learned_addr learned my addr 172.21.15.86:0/406479810 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T10:34:52.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 1 -- 172.21.15.86:0/406479810 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3665232348 0 0) 0x7fa89809da30 con 0x7fa89809c870 2024-03-29T10:34:52.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:34:52.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:52.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:52.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 10 monclient(hunting): my global_id is 4132 2024-03-29T10:34:52.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 1 -- 172.21.15.86:0/406479810 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa884003520 con 0x7fa89809c870 2024-03-29T10:34:52.823 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 1 -- 172.21.15.86:0/406479810 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3979019541 0 0) 0x7fa888001110 con 0x7fa89809ac40 2024-03-29T10:34:52.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:34:52.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:52.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:52.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 10 monclient(hunting): my global_id is 4140 2024-03-29T10:34:52.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 1 -- 172.21.15.86:0/406479810 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa89809da30 con 0x7fa89809ac40 2024-03-29T10:34:52.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 1 -- 172.21.15.86:0/406479810 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3620398571 0 0) 0x7fa8980b39c0 con 0x7fa8980b4360 2024-03-29T10:34:52.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:34:52.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:52.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:52.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 10 monclient(hunting): my global_id is 4136 2024-03-29T10:34:52.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 1 -- 172.21.15.86:0/406479810 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa888001110 con 0x7fa8980b4360 2024-03-29T10:34:52.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 1 -- 172.21.15.86:0/406479810 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1749834209 0 0) 0x7fa884003520 con 0x7fa89809c870 2024-03-29T10:34:52.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 1 -- 172.21.15.86:0/406479810 >> v1:172.21.15.84:6790/0 conn(0x7fa8980b4360 legacy=0x7fa89814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 1 -- 172.21.15.86:0/406479810 >> v1:172.21.15.84:6789/0 conn(0x7fa89809ac40 legacy=0x7fa8980b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.824 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:34:52.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 1 monclient: found mon.noname-c 2024-03-29T10:34:52.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:52.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-29T10:34:52.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 1 -- 172.21.15.86:0/406479810 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8980b5160 con 0x7fa89809c870 2024-03-29T10:34:52.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 10 monclient: _finish_auth 0 2024-03-29T10:34:52.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:34:52.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:52.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 1 -- 172.21.15.86:0/406479810 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa894001690 con 0x7fa89809c870 2024-03-29T10:34:52.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:52.825 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 10 monclient: dump: 2024-03-29T10:34:52.825 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.825 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.825 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.825 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.826 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.826 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.826 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.826 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.826 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.826 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa89f9d6640 5 monclient: authenticate success, global_id 4132 2024-03-29T10:34:52.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa89f9d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:52.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.828+0000 7fa88ffff640 1 -- 172.21.15.86:0/406479810 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa894001870 con 0x7fa89809c870 2024-03-29T10:34:52.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.829+0000 7fa88ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:52.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.829+0000 7fa88ffff640 1 -- 172.21.15.86:0/406479810 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa894001c80 con 0x7fa89809c870 2024-03-29T10:34:52.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.829+0000 7fa89f9d6640 10 monclient: get_monmap_and_config success 2024-03-29T10:34:52.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.829+0000 7fa89f9d6640 4 set_mon_vals no callback set 2024-03-29T10:34:52.826 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.829+0000 7fa88ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.829+0000 7fa88ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:52.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.829+0000 7fa88ffff640 10 monclient: dump: 2024-03-29T10:34:52.827 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.827 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.827 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.827 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.827 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.827 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.827 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.827 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.827 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.827 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.829+0000 7fa89f9d6640 10 monclient: shutdown 2024-03-29T10:34:52.827 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.829+0000 7fa89f9d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:52.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.829+0000 7fa89f9d6640 1 -- 172.21.15.86:0/406479810 >> v1:172.21.15.86:6789/0 conn(0x7fa89809c870 legacy=0x7fa89813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.829+0000 7fa89f9d6640 1 -- 172.21.15.86:0/406479810 shutdown_connections 2024-03-29T10:34:52.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.829+0000 7fa89f9d6640 1 -- 172.21.15.86:0/406479810 >> 172.21.15.86:0/406479810 conn(0x7fa898066b00 msgr2=0x7fa898066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:52.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.829+0000 7fa89f9d6640 1 -- 172.21.15.86:0/406479810 shutdown_connections 2024-03-29T10:34:52.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.829+0000 7fa89f9d6640 1 -- 172.21.15.86:0/406479810 wait complete. 2024-03-29T10:34:52.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.830+0000 7fa89f9d6640 10 monclient: build_initial_monmap 2024-03-29T10:34:52.828 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.830+0000 7fa89f9d6640 10 monclient: monmap: 2024-03-29T10:34:52.828 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-29T10:34:52.828 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:34:52.828 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-03-29T10:34:52.828 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-03-29T10:34:52.828 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:52.829 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.829 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:34:52.829 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:34:52.829 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:34:52.829 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.831+0000 7fa89f9d6640 1 librados: starting msgr at 2024-03-29T10:34:52.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.831+0000 7fa89f9d6640 1 librados: starting objecter 2024-03-29T10:34:52.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.831+0000 7fa89f9d6640 1 Processor -- start 2024-03-29T10:34:52.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.831+0000 7fa89f9d6640 1 -- start start 2024-03-29T10:34:52.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.831+0000 7fa89f9d6640 1 librados: setting wanted keys 2024-03-29T10:34:52.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.831+0000 7fa89f9d6640 1 librados: calling monclient init 2024-03-29T10:34:52.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.831+0000 7fa89f9d6640 10 monclient: init 2024-03-29T10:34:52.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa89f9d6640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:52.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa89f9d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:34:52.829 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa89f9d6640 10 monclient(hunting): picked mon.noname-a con 0x7fa89809ac40 addr v1:172.21.15.84:6789/0 2024-03-29T10:34:52.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa89f9d6640 10 monclient(hunting): picked mon.noname-b con 0x7fa8981d7bc0 addr v1:172.21.15.86:6789/0 2024-03-29T10:34:52.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa89f9d6640 10 monclient(hunting): picked mon.noname-c con 0x7fa8980b17f0 addr v1:172.21.15.84:6790/0 2024-03-29T10:34:52.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa89f9d6640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa89813b890 con 0x7fa89809ac40 2024-03-29T10:34:52.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa89f9d6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8981d76a0 con 0x7fa8981d7bc0 2024-03-29T10:34:52.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa89f9d6640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8981406b0 con 0x7fa8980b17f0 2024-03-29T10:34:52.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa89f9d6640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:52.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa89f9d6640 10 monclient(hunting): authenticate will time out at 776.997532s 2024-03-29T10:34:52.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa89cf71640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa8981d7bc0 0x7fa89813aa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:50330/0 (socket says 172.21.15.86:50330) 2024-03-29T10:34:52.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa89cf71640 1 -- 172.21.15.86:0/3294261235 learned_addr learned my addr 172.21.15.86:0/3294261235 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T10:34:52.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa88dffb640 10 client.?.objecter ms_handle_connect 0x7fa8981d7bc0 2024-03-29T10:34:52.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa88dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:52.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa88dffb640 10 client.?.objecter ms_handle_connect 0x7fa89809ac40 2024-03-29T10:34:52.830 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa88dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:52.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa88dffb640 10 client.?.objecter ms_handle_connect 0x7fa8980b17f0 2024-03-29T10:34:52.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa88dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:52.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa88dffb640 1 -- 172.21.15.86:0/3294261235 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3869534437 0 0) 0x7fa8981d76a0 con 0x7fa8981d7bc0 2024-03-29T10:34:52.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa88dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:34:52.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa88dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:52.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa88dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:52.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa88dffb640 10 monclient(hunting): my global_id is 4135 2024-03-29T10:34:52.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa88dffb640 1 -- 172.21.15.86:0/3294261235 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa87c004eb0 con 0x7fa8981d7bc0 2024-03-29T10:34:52.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa88dffb640 1 -- 172.21.15.86:0/3294261235 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1375674292 0 0) 0x7fa89813b890 con 0x7fa89809ac40 2024-03-29T10:34:52.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa88dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:34:52.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa88dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:52.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa88dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:52.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa88dffb640 10 monclient(hunting): my global_id is 4143 2024-03-29T10:34:52.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa88dffb640 1 -- 172.21.15.86:0/3294261235 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8981d76a0 con 0x7fa89809ac40 2024-03-29T10:34:52.831 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa88dffb640 1 -- 172.21.15.86:0/3294261235 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 509949390 0 0) 0x7fa8981406b0 con 0x7fa8980b17f0 2024-03-29T10:34:52.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa88dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:34:52.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa88dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:52.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa88dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:52.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa88dffb640 10 monclient(hunting): my global_id is 4139 2024-03-29T10:34:52.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa88dffb640 1 -- 172.21.15.86:0/3294261235 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa89813b890 con 0x7fa8980b17f0 2024-03-29T10:34:52.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa88dffb640 1 -- 172.21.15.86:0/3294261235 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1206259156 0 0) 0x7fa87c004eb0 con 0x7fa8981d7bc0 2024-03-29T10:34:52.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa88dffb640 1 -- 172.21.15.86:0/3294261235 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8981406b0 con 0x7fa8981d7bc0 2024-03-29T10:34:52.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa88dffb640 1 -- 172.21.15.86:0/3294261235 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa894001bc0 con 0x7fa8981d7bc0 2024-03-29T10:34:52.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.832+0000 7fa88dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:34:52.832 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 10 monclient(hunting): dump: 2024-03-29T10:34:52.832 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.832 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.832 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.832 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.833 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.833 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.833 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.833 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.833 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.833 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:34:52.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 1 -- 172.21.15.86:0/3294261235 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2760399048 0 0) 0x7fa8981d76a0 con 0x7fa89809ac40 2024-03-29T10:34:52.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 1 -- 172.21.15.86:0/3294261235 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa87c004eb0 con 0x7fa89809ac40 2024-03-29T10:34:52.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 1 -- 172.21.15.86:0/3294261235 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa8880018a0 con 0x7fa89809ac40 2024-03-29T10:34:52.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:34:52.833 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 10 monclient(hunting): dump: 2024-03-29T10:34:52.833 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.834 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.834 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.834 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.834 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.834 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.834 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.834 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.834 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.834 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 1 -- 172.21.15.86:0/3294261235 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 515997722 0 0) 0x7fa89813b890 con 0x7fa8980b17f0 2024-03-29T10:34:52.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 1 -- 172.21.15.86:0/3294261235 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8981d76a0 con 0x7fa8980b17f0 2024-03-29T10:34:52.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 1 -- 172.21.15.86:0/3294261235 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa890001b50 con 0x7fa8980b17f0 2024-03-29T10:34:52.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.834 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:34:52.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 10 monclient(hunting): dump: 2024-03-29T10:34:52.835 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.835 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.835 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.835 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.835 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.835 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.835 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.835 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.835 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.835 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 1 -- 172.21.15.86:0/3294261235 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1189739847 0 0) 0x7fa8981406b0 con 0x7fa8981d7bc0 2024-03-29T10:34:52.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 1 -- 172.21.15.86:0/3294261235 >> v1:172.21.15.84:6790/0 conn(0x7fa8980b17f0 legacy=0x7fa89813b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 1 -- 172.21.15.86:0/3294261235 >> v1:172.21.15.84:6789/0 conn(0x7fa89809ac40 legacy=0x7fa89813ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.835 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:34:52.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 1 monclient: found mon.b 2024-03-29T10:34:52.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:52.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:52.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 1 -- 172.21.15.86:0/3294261235 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa898140840 con 0x7fa8981d7bc0 2024-03-29T10:34:52.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 10 monclient: _finish_auth 0 2024-03-29T10:34:52.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:34:52.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:52.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa89f9d6640 5 monclient: authenticate success, global_id 4135 2024-03-29T10:34:52.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa89f9d6640 10 monclient: _renew_subs 2024-03-29T10:34:52.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa89f9d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:52.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa89f9d6640 1 -- 172.21.15.86:0/3294261235 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8980b6f40 con 0x7fa8981d7bc0 2024-03-29T10:34:52.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 1 -- 172.21.15.86:0/3294261235 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa894004bf0 con 0x7fa8981d7bc0 2024-03-29T10:34:52.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:52.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 1 -- 172.21.15.86:0/3294261235 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa894004f00 con 0x7fa8981d7bc0 2024-03-29T10:34:52.836 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:52.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:52.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 10 monclient: dump: 2024-03-29T10:34:52.837 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:52.837 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:52.837 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.837 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:52.837 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:52.837 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:52.837 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:52.837 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:52.837 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:52.837 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:52.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 1 -- 172.21.15.86:0/3294261235 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa894005470 con 0x7fa8981d7bc0 2024-03-29T10:34:52.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88f7fe640 4 set_mon_vals no callback set 2024-03-29T10:34:52.837 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa89f9d6640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:34:52.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa89f9d6640 10 monclient: _renew_subs 2024-03-29T10:34:52.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa89f9d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:52.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa89f9d6640 1 -- 172.21.15.86:0/3294261235 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8981d6f40 con 0x7fa8981d7bc0 2024-03-29T10:34:52.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa89f9d6640 1 librados: init done 2024-03-29T10:34:52.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 1 -- 172.21.15.86:0/3294261235 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 872327613 0 0) 0x7fa894005eb0 con 0x7fa8981d7bc0 2024-03-29T10:34:52.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 10 client.4135.objecter ms_dispatch 0x7fa898000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T10:34:52.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 3 client.4135.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T10:34:52.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 3 client.4135.objecter handle_osd_map decoding full epoch 6 2024-03-29T10:34:52.838 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.833+0000 7fa88dffb640 20 client.4135.objecter dump_active .. 0 homeless 2024-03-29T10:34:52.838 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-29T10:34:52.834+0000 7f4e58c21040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:34:52.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.834+0000 7fa89f9d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:52.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.834+0000 7fa89f9d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:52.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.834+0000 7fa89f9d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:52.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.834+0000 7fa89f9d6640 1 -- 172.21.15.86:0/3294261235 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa89809b020 con 0x7fa8981d7bc0 2024-03-29T10:34:52.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.836+0000 7fa88dffb640 1 -- 172.21.15.86:0/3294261235 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa894006070 con 0x7fa8981d7bc0 2024-03-29T10:34:52.839 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.836+0000 7fa88dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:52.840 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.836+0000 7fa88dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:34:52.853 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T10:34:52.867+0000 7f187bc42040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:34:52.866 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-29T10:34:52.881+0000 7f4e58c21040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:34:52.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.906+0000 7fa89f9d6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0c8cbbef-4bb9-475f-b150-0e97365d0d56", "id": 4}] 2024-03-29T10:34:52.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.906+0000 7fa89f9d6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0c8cbbef-4bb9-475f-b150-0e97365d0d56", "id": 4}] 2024-03-29T10:34:52.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.906+0000 7fa89f9d6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:52.892 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.906+0000 7fa89f9d6640 1 -- 172.21.15.86:0/3294261235 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd new", "uuid": "0c8cbbef-4bb9-475f-b150-0e97365d0d56", "id": 4} v 0) v1 -- 0x7fa89804f730 con 0x7fa8981d7bc0 2024-03-29T10:34:52.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.910+0000 7fa88dffb640 1 -- 172.21.15.86:0/3294261235 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0c8cbbef-4bb9-475f-b150-0e97365d0d56", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 3255472467 0 1697012583) 0x7fa8940059a0 con 0x7fa8981d7bc0 2024-03-29T10:34:52.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.910+0000 7fa88dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0c8cbbef-4bb9-475f-b150-0e97365d0d56", "id": 4}] 2024-03-29T10:34:52.896 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.910+0000 7fa88dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:34:52.896 INFO:teuthology.orchestra.run.smithi086.stdout:4 2024-03-29T10:34:52.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.911+0000 7fa89f9d6640 10 librados: watch_flush enter 2024-03-29T10:34:52.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.911+0000 7fa89f9d6640 10 librados: watch_flush exit 2024-03-29T10:34:52.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.911+0000 7fa89f9d6640 20 client.4135.objecter shutdown clearing up homeless session... 2024-03-29T10:34:52.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.911+0000 7fa89f9d6640 10 client.4135.objecter successfully canceled tick 2024-03-29T10:34:52.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.911+0000 7fa89f9d6640 10 monclient: shutdown 2024-03-29T10:34:52.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.911+0000 7fa89f9d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:52.898 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.911+0000 7fa89f9d6640 1 -- 172.21.15.86:0/3294261235 >> v1:172.21.15.86:6789/0 conn(0x7fa8981d7bc0 legacy=0x7fa89813aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:52.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.911+0000 7fa89f9d6640 1 -- 172.21.15.86:0/3294261235 shutdown_connections 2024-03-29T10:34:52.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.911+0000 7fa89f9d6640 1 -- 172.21.15.86:0/3294261235 >> 172.21.15.86:0/3294261235 conn(0x7fa898066b00 msgr2=0x7fa89813d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:52.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.912+0000 7fa89f9d6640 1 -- 172.21.15.86:0/3294261235 shutdown_connections 2024-03-29T10:34:52.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.912+0000 7fa89f9d6640 1 -- 172.21.15.86:0/3294261235 wait complete. 2024-03-29T10:34:52.899 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:52.912+0000 7fa89f9d6640 1 librados: shutdown 2024-03-29T10:34:52.907 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 93c3ca84-cf2a-4741-b3d1-ca994791a7c5 5 2024-03-29T10:34:52.907 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T10:34:52.921+0000 7f187bc42040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:34:52.920 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-29T10:34:52.935+0000 7f4e58c21040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:34:52.954 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T10:34:52.968+0000 7f187bc42040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:34:52.970 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-29T10:34:52.985+0000 7f4e58c21040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:34:53.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.033+0000 7f65142c5640 10 monclient: get_monmap_and_config 2024-03-29T10:34:53.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.033+0000 7f65142c5640 10 monclient: build_initial_monmap 2024-03-29T10:34:53.019 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.033+0000 7f65142c5640 10 monclient: monmap: 2024-03-29T10:34:53.019 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-29T10:34:53.019 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:53.019 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:53.034863+0000 2024-03-29T10:34:53.019 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:53.034863+0000 2024-03-29T10:34:53.019 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:53.019 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:53.019 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:34:53.019 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:34:53.020 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:34:53.020 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:53.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.034+0000 7f65142c5640 1 Processor -- start 2024-03-29T10:34:53.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.034+0000 7f65142c5640 1 -- start start 2024-03-29T10:34:53.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.034+0000 7f65142c5640 10 monclient: init 2024-03-29T10:34:53.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f65142c5640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:53.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f65142c5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:34:53.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f65142c5640 10 monclient(hunting): picked mon.noname-b con 0x7f650c09ac40 addr v1:172.21.15.84:6790/0 2024-03-29T10:34:53.027 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f65142c5640 10 monclient(hunting): picked mon.noname-a con 0x7f650c09c870 addr v1:172.21.15.84:6789/0 2024-03-29T10:34:53.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f65142c5640 10 monclient(hunting): picked mon.noname-c con 0x7f650c0b4360 addr v1:172.21.15.86:6789/0 2024-03-29T10:34:53.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f65142c5640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f650c09dec0 con 0x7f650c09c870 2024-03-29T10:34:53.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f65142c5640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f650c09da30 con 0x7f650c0b4360 2024-03-29T10:34:53.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f65142c5640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f650c0b39c0 con 0x7f650c09ac40 2024-03-29T10:34:53.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f65142c5640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:53.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f65142c5640 10 monclient(hunting): authenticate will time out at 777.200266s 2024-03-29T10:34:53.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f6512862640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f650c0b4360 0x7f650c14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:50338/0 (socket says 172.21.15.86:50338) 2024-03-29T10:34:53.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f6512862640 1 -- 172.21.15.86:0/4278934388 learned_addr learned my addr 172.21.15.86:0/4278934388 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T10:34:53.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 1 -- 172.21.15.86:0/4278934388 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849583755 0 0) 0x7f650c09da30 con 0x7f650c0b4360 2024-03-29T10:34:53.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 10 monclient(hunting): _negotiate 2024-03-29T10:34:53.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:53.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:53.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 10 monclient(hunting): my global_id is 4138 2024-03-29T10:34:53.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 1 -- 172.21.15.86:0/4278934388 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64f8003520 con 0x7f650c0b4360 2024-03-29T10:34:53.028 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 1 -- 172.21.15.86:0/4278934388 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 218467737 0 0) 0x7f650c0b39c0 con 0x7f650c09ac40 2024-03-29T10:34:53.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 10 monclient(hunting): _negotiate 2024-03-29T10:34:53.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:53.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:53.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 10 monclient(hunting): my global_id is 4142 2024-03-29T10:34:53.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 1 -- 172.21.15.86:0/4278934388 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f650c09da30 con 0x7f650c09ac40 2024-03-29T10:34:53.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 1 -- 172.21.15.86:0/4278934388 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2685329848 0 0) 0x7f6508001110 con 0x7f650c09c870 2024-03-29T10:34:53.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 10 monclient(hunting): _negotiate 2024-03-29T10:34:53.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:53.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:53.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 10 monclient(hunting): my global_id is 4146 2024-03-29T10:34:53.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 1 -- 172.21.15.86:0/4278934388 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f650c0b39c0 con 0x7f650c09c870 2024-03-29T10:34:53.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 1 -- 172.21.15.86:0/4278934388 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1947358605 0 0) 0x7f64f8003520 con 0x7f650c0b4360 2024-03-29T10:34:53.029 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 1 -- 172.21.15.86:0/4278934388 >> v1:172.21.15.84:6790/0 conn(0x7f650c09ac40 legacy=0x7f650c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:53.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 1 -- 172.21.15.86:0/4278934388 >> v1:172.21.15.84:6789/0 conn(0x7f650c09c870 legacy=0x7f650c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:53.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 10 monclient: _finish_hunting 0 2024-03-29T10:34:53.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 1 monclient: found mon.noname-c 2024-03-29T10:34:53.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:53.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-29T10:34:53.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 1 -- 172.21.15.86:0/4278934388 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f650c0b5160 con 0x7f650c0b4360 2024-03-29T10:34:53.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 10 monclient: _finish_auth 0 2024-03-29T10:34:53.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 10 monclient: _check_auth_tickets 2024-03-29T10:34:53.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:53.030 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f65142c5640 5 monclient: authenticate success, global_id 4138 2024-03-29T10:34:53.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 1 -- 172.21.15.86:0/4278934388 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f6504001a10 con 0x7f650c0b4360 2024-03-29T10:34:53.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f65142c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:53.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:53.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:53.031 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 10 monclient: dump: 2024-03-29T10:34:53.031 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:53.031 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:53.031 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.031 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.031 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:53.031 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:53.031 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:53.031 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:53.032 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:53.032 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:53.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f65142c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:53.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 1 -- 172.21.15.86:0/4278934388 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6504001bf0 con 0x7f650c0b4360 2024-03-29T10:34:53.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:53.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f65142c5640 10 monclient: get_monmap_and_config success 2024-03-29T10:34:53.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 1 -- 172.21.15.86:0/4278934388 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f6504002000 con 0x7f650c0b4360 2024-03-29T10:34:53.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f65142c5640 4 set_mon_vals no callback set 2024-03-29T10:34:53.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:53.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:53.032 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.035+0000 7f651105f640 10 monclient: dump: 2024-03-29T10:34:53.032 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:53.032 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:53.032 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.032 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.033 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:53.033 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:53.033 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:53.033 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:53.033 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:53.033 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:53.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.036+0000 7f65142c5640 10 monclient: shutdown 2024-03-29T10:34:53.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.036+0000 7f65142c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:53.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.036+0000 7f65142c5640 1 -- 172.21.15.86:0/4278934388 >> v1:172.21.15.86:6789/0 conn(0x7f650c0b4360 legacy=0x7f650c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:53.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.036+0000 7f65142c5640 1 -- 172.21.15.86:0/4278934388 shutdown_connections 2024-03-29T10:34:53.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.036+0000 7f65142c5640 1 -- 172.21.15.86:0/4278934388 >> 172.21.15.86:0/4278934388 conn(0x7f650c066b00 msgr2=0x7f650c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:53.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.036+0000 7f65142c5640 1 -- 172.21.15.86:0/4278934388 shutdown_connections 2024-03-29T10:34:53.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.036+0000 7f65142c5640 1 -- 172.21.15.86:0/4278934388 wait complete. 2024-03-29T10:34:53.033 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.037+0000 7f65142c5640 10 monclient: build_initial_monmap 2024-03-29T10:34:53.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.037+0000 7f65142c5640 10 monclient: monmap: 2024-03-29T10:34:53.034 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-29T10:34:53.034 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:34:53.034 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-03-29T10:34:53.034 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-03-29T10:34:53.034 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:53.034 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:53.034 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:34:53.034 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:34:53.034 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:34:53.034 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:53.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.037+0000 7f65142c5640 1 librados: starting msgr at 2024-03-29T10:34:53.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.037+0000 7f65142c5640 1 librados: starting objecter 2024-03-29T10:34:53.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.037+0000 7f65142c5640 1 Processor -- start 2024-03-29T10:34:53.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.037+0000 7f65142c5640 1 -- start start 2024-03-29T10:34:53.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.037+0000 7f65142c5640 1 librados: setting wanted keys 2024-03-29T10:34:53.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.037+0000 7f65142c5640 1 librados: calling monclient init 2024-03-29T10:34:53.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.037+0000 7f65142c5640 10 monclient: init 2024-03-29T10:34:53.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.037+0000 7f65142c5640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:53.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f65142c5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:34:53.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f65142c5640 10 monclient(hunting): picked mon.noname-b con 0x7f650c09ac40 addr v1:172.21.15.86:6789/0 2024-03-29T10:34:53.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f65142c5640 10 monclient(hunting): picked mon.noname-a con 0x7f650c0b7310 addr v1:172.21.15.84:6789/0 2024-03-29T10:34:53.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f65142c5640 10 monclient(hunting): picked mon.noname-c con 0x7f650c1d7de0 addr v1:172.21.15.84:6790/0 2024-03-29T10:34:53.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f65142c5640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f650c0b78c0 con 0x7f650c0b7310 2024-03-29T10:34:53.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f65142c5640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f650c0b7d50 con 0x7f650c09ac40 2024-03-29T10:34:53.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f65142c5640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f650c146480 con 0x7f650c1d7de0 2024-03-29T10:34:53.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f65142c5640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:53.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f65142c5640 10 monclient(hunting): authenticate will time out at 777.203154s 2024-03-29T10:34:53.035 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f6512061640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f650c09ac40 0x7f650c0b6a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:50350/0 (socket says 172.21.15.86:50350) 2024-03-29T10:34:53.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f6512061640 1 -- 172.21.15.86:0/1351478797 learned_addr learned my addr 172.21.15.86:0/1351478797 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T10:34:53.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 client.?.objecter ms_handle_connect 0x7f650c09ac40 2024-03-29T10:34:53.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:53.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 client.?.objecter ms_handle_connect 0x7f650c0b7310 2024-03-29T10:34:53.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:53.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 client.?.objecter ms_handle_connect 0x7f650c1d7de0 2024-03-29T10:34:53.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:53.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 1 -- 172.21.15.86:0/1351478797 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359825113 0 0) 0x7f650c0b7d50 con 0x7f650c09ac40 2024-03-29T10:34:53.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:53.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:53.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:53.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 monclient(hunting): my global_id is 4141 2024-03-29T10:34:53.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 1 -- 172.21.15.86:0/1351478797 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64f0004eb0 con 0x7f650c09ac40 2024-03-29T10:34:53.036 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 1 -- 172.21.15.86:0/1351478797 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3847391488 0 0) 0x7f650c0b78c0 con 0x7f650c0b7310 2024-03-29T10:34:53.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:53.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:53.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:53.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 monclient(hunting): my global_id is 4149 2024-03-29T10:34:53.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 1 -- 172.21.15.86:0/1351478797 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f650c0b7d50 con 0x7f650c0b7310 2024-03-29T10:34:53.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 1 -- 172.21.15.86:0/1351478797 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452491121 0 0) 0x7f650c146480 con 0x7f650c1d7de0 2024-03-29T10:34:53.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:53.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:53.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:53.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 monclient(hunting): my global_id is 4145 2024-03-29T10:34:53.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 1 -- 172.21.15.86:0/1351478797 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f650c0b78c0 con 0x7f650c1d7de0 2024-03-29T10:34:53.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 1 -- 172.21.15.86:0/1351478797 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3748196957 0 0) 0x7f64f0004eb0 con 0x7f650c09ac40 2024-03-29T10:34:53.037 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 1 -- 172.21.15.86:0/1351478797 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f650c146480 con 0x7f650c09ac40 2024-03-29T10:34:53.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 1 -- 172.21.15.86:0/1351478797 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f65000019e0 con 0x7f650c09ac40 2024-03-29T10:34:53.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:53.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:34:53.038 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 monclient(hunting): dump: 2024-03-29T10:34:53.038 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:53.038 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:53.038 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.038 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.038 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:53.038 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:53.038 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:53.038 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:53.038 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:53.038 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:53.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:34:53.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 1 -- 172.21.15.86:0/1351478797 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4199996957 0 0) 0x7f650c0b7d50 con 0x7f650c0b7310 2024-03-29T10:34:53.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 1 -- 172.21.15.86:0/1351478797 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64f0004eb0 con 0x7f650c0b7310 2024-03-29T10:34:53.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 1 -- 172.21.15.86:0/1351478797 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f6508001760 con 0x7f650c0b7310 2024-03-29T10:34:53.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:53.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:34:53.039 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 monclient(hunting): dump: 2024-03-29T10:34:53.039 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:53.039 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:53.039 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.039 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.039 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:53.039 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:53.039 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:53.040 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:53.040 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:53.040 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:53.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 1 -- 172.21.15.86:0/1351478797 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1118757304 0 0) 0x7f650c0b78c0 con 0x7f650c1d7de0 2024-03-29T10:34:53.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 1 -- 172.21.15.86:0/1351478797 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f650c0b7d50 con 0x7f650c1d7de0 2024-03-29T10:34:53.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 1 -- 172.21.15.86:0/1351478797 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f6504001f00 con 0x7f650c1d7de0 2024-03-29T10:34:53.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:53.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:34:53.040 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 monclient(hunting): dump: 2024-03-29T10:34:53.040 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:53.040 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:53.040 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.040 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.040 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:53.041 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:53.041 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:53.041 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:53.041 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:53.041 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:53.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 1 -- 172.21.15.86:0/1351478797 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2011240808 0 0) 0x7f650c146480 con 0x7f650c09ac40 2024-03-29T10:34:53.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 1 -- 172.21.15.86:0/1351478797 >> v1:172.21.15.84:6790/0 conn(0x7f650c1d7de0 legacy=0x7f650c1e2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:53.041 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 1 -- 172.21.15.86:0/1351478797 >> v1:172.21.15.84:6789/0 conn(0x7f650c0b7310 legacy=0x7f650c1d76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:53.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 monclient: _finish_hunting 0 2024-03-29T10:34:53.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 1 monclient: found mon.b 2024-03-29T10:34:53.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:53.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:53.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 1 -- 172.21.15.86:0/1351478797 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f650c1e68d0 con 0x7f650c09ac40 2024-03-29T10:34:53.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 monclient: _finish_auth 0 2024-03-29T10:34:53.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 monclient: _check_auth_tickets 2024-03-29T10:34:53.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:53.042 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f65142c5640 5 monclient: authenticate success, global_id 4141 2024-03-29T10:34:53.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f65142c5640 10 monclient: _renew_subs 2024-03-29T10:34:53.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f65142c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:53.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f65142c5640 1 -- 172.21.15.86:0/1351478797 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f650c1e38a0 con 0x7f650c09ac40 2024-03-29T10:34:53.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f65142c5640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:34:53.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f65142c5640 10 monclient: _renew_subs 2024-03-29T10:34:53.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f65142c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:53.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f65142c5640 1 -- 172.21.15.86:0/1351478797 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f650c1e3dd0 con 0x7f650c09ac40 2024-03-29T10:34:53.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f65142c5640 1 librados: init done 2024-03-29T10:34:53.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 1 -- 172.21.15.86:0/1351478797 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6500001c80 con 0x7f650c09ac40 2024-03-29T10:34:53.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:53.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 1 -- 172.21.15.86:0/1351478797 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f6500003a60 con 0x7f650c09ac40 2024-03-29T10:34:53.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.038+0000 7f64feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:53.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.039+0000 7f64feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:53.043 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.039+0000 7f64feffd640 10 monclient: dump: 2024-03-29T10:34:53.043 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:53.044 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:53.044 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.044 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.044 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:53.044 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:53.044 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:53.044 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:53.044 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:53.044 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:53.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.039+0000 7f651085e640 4 set_mon_vals no callback set 2024-03-29T10:34:53.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.039+0000 7f64feffd640 1 -- 172.21.15.86:0/1351478797 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6500004010 con 0x7f650c09ac40 2024-03-29T10:34:53.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.039+0000 7f64feffd640 1 -- 172.21.15.86:0/1351478797 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 3734750104 0 0) 0x7f6500006920 con 0x7f650c09ac40 2024-03-29T10:34:53.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.039+0000 7f64feffd640 10 client.4141.objecter ms_dispatch 0x7f650c000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T10:34:53.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.039+0000 7f64feffd640 3 client.4141.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T10:34:53.044 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.039+0000 7f64feffd640 3 client.4141.objecter handle_osd_map decoding full epoch 7 2024-03-29T10:34:53.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.039+0000 7f64feffd640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-29T10:34:53.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.039+0000 7f65142c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:53.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.039+0000 7f65142c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:53.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.039+0000 7f65142c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:53.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.039+0000 7f65142c5640 1 -- 172.21.15.86:0/1351478797 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64f4005080 con 0x7f650c09ac40 2024-03-29T10:34:53.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.041+0000 7f64feffd640 1 -- 172.21.15.86:0/1351478797 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f65000063a0 con 0x7f650c09ac40 2024-03-29T10:34:53.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.041+0000 7f64feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:53.045 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.041+0000 7f64feffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:34:53.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.113+0000 7f65142c5640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "93c3ca84-cf2a-4741-b3d1-ca994791a7c5", "id": 5}] 2024-03-29T10:34:53.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.113+0000 7f65142c5640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "93c3ca84-cf2a-4741-b3d1-ca994791a7c5", "id": 5}] 2024-03-29T10:34:53.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.113+0000 7f65142c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:53.099 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.113+0000 7f65142c5640 1 -- 172.21.15.86:0/1351478797 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd new", "uuid": "93c3ca84-cf2a-4741-b3d1-ca994791a7c5", "id": 5} v 0) v1 -- 0x7f64f40058b0 con 0x7f650c09ac40 2024-03-29T10:34:53.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.117+0000 7f64feffd640 1 -- 172.21.15.86:0/1351478797 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "93c3ca84-cf2a-4741-b3d1-ca994791a7c5", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3435322521 0 1988412176) 0x7f6500006600 con 0x7f650c09ac40 2024-03-29T10:34:53.102 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.117+0000 7f64feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "93c3ca84-cf2a-4741-b3d1-ca994791a7c5", "id": 5}] 2024-03-29T10:34:53.103 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.117+0000 7f64feffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:34:53.103 INFO:teuthology.orchestra.run.smithi086.stdout:5 2024-03-29T10:34:53.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.118+0000 7f65142c5640 10 librados: watch_flush enter 2024-03-29T10:34:53.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.118+0000 7f65142c5640 10 librados: watch_flush exit 2024-03-29T10:34:53.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.118+0000 7f65142c5640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-29T10:34:53.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.118+0000 7f65142c5640 10 client.4141.objecter successfully canceled tick 2024-03-29T10:34:53.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.118+0000 7f65142c5640 10 monclient: shutdown 2024-03-29T10:34:53.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.118+0000 7f65142c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:53.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.118+0000 7f65142c5640 1 -- 172.21.15.86:0/1351478797 >> v1:172.21.15.86:6789/0 conn(0x7f650c09ac40 legacy=0x7f650c0b6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:53.104 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.118+0000 7f65142c5640 1 -- 172.21.15.86:0/1351478797 shutdown_connections 2024-03-29T10:34:53.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.118+0000 7f65142c5640 1 -- 172.21.15.86:0/1351478797 >> 172.21.15.86:0/1351478797 conn(0x7f650c066b00 msgr2=0x7f650c13a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:53.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.118+0000 7f65142c5640 1 -- 172.21.15.86:0/1351478797 shutdown_connections 2024-03-29T10:34:53.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.118+0000 7f65142c5640 1 -- 172.21.15.86:0/1351478797 wait complete. 2024-03-29T10:34:53.105 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.118+0000 7f65142c5640 1 librados: shutdown 2024-03-29T10:34:53.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new a6752d73-9b2c-451f-92fe-1baa4b29cc9a 6 2024-03-29T10:34:53.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.228+0000 7f8101aff640 10 monclient: get_monmap_and_config 2024-03-29T10:34:53.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.228+0000 7f8101aff640 10 monclient: build_initial_monmap 2024-03-29T10:34:53.213 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.228+0000 7f8101aff640 10 monclient: monmap: 2024-03-29T10:34:53.214 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-29T10:34:53.214 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:53.214 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:53.229499+0000 2024-03-29T10:34:53.214 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:53.229499+0000 2024-03-29T10:34:53.214 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:53.214 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:53.214 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:34:53.214 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:34:53.214 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:34:53.214 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:53.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.229+0000 7f8101aff640 1 Processor -- start 2024-03-29T10:34:53.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.229+0000 7f8101aff640 1 -- start start 2024-03-29T10:34:53.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.229+0000 7f8101aff640 10 monclient: init 2024-03-29T10:34:53.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.229+0000 7f8101aff640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:53.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.229+0000 7f8101aff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:34:53.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.229+0000 7f8101aff640 10 monclient(hunting): picked mon.noname-b con 0x7f80fc09ac40 addr v1:172.21.15.84:6790/0 2024-03-29T10:34:53.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.229+0000 7f8101aff640 10 monclient(hunting): picked mon.noname-c con 0x7f80fc09c870 addr v1:172.21.15.86:6789/0 2024-03-29T10:34:53.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.229+0000 7f8101aff640 10 monclient(hunting): picked mon.noname-a con 0x7f80fc0b4360 addr v1:172.21.15.84:6789/0 2024-03-29T10:34:53.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.229+0000 7f8101aff640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80fc09dec0 con 0x7f80fc0b4360 2024-03-29T10:34:53.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.229+0000 7f8101aff640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80fc09da30 con 0x7f80fc09c870 2024-03-29T10:34:53.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.229+0000 7f8101aff640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80fc0b39c0 con 0x7f80fc09ac40 2024-03-29T10:34:53.223 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.229+0000 7f8101aff640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:53.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.229+0000 7f8101aff640 10 monclient(hunting): authenticate will time out at 777.394880s 2024-03-29T10:34:53.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.229+0000 7f80faffd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f80fc09c870 0x7f80fc13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:50354/0 (socket says 172.21.15.86:50354) 2024-03-29T10:34:53.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.229+0000 7f80faffd640 1 -- 172.21.15.86:0/2361528006 learned_addr learned my addr 172.21.15.86:0/2361528006 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T10:34:53.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 1 -- 172.21.15.86:0/2361528006 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22558530 0 0) 0x7f80fc09da30 con 0x7f80fc09c870 2024-03-29T10:34:53.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:34:53.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:53.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:53.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 10 monclient(hunting): my global_id is 4144 2024-03-29T10:34:53.224 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 1 -- 172.21.15.86:0/2361528006 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80e4003520 con 0x7f80fc09c870 2024-03-29T10:34:53.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 1 -- 172.21.15.86:0/2361528006 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1464461568 0 0) 0x7f80fc0b39c0 con 0x7f80fc09ac40 2024-03-29T10:34:53.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:34:53.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:53.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:53.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 10 monclient(hunting): my global_id is 4148 2024-03-29T10:34:53.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 1 -- 172.21.15.86:0/2361528006 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80fc09da30 con 0x7f80fc09ac40 2024-03-29T10:34:53.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 1 -- 172.21.15.86:0/2361528006 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021137646 0 0) 0x7f80f4001490 con 0x7f80fc0b4360 2024-03-29T10:34:53.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:34:53.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:53.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:53.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 10 monclient(hunting): my global_id is 4152 2024-03-29T10:34:53.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 1 -- 172.21.15.86:0/2361528006 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80fc0b39c0 con 0x7f80fc0b4360 2024-03-29T10:34:53.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 1 -- 172.21.15.86:0/2361528006 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3864273716 0 0) 0x7f80e4003520 con 0x7f80fc09c870 2024-03-29T10:34:53.225 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 1 -- 172.21.15.86:0/2361528006 >> v1:172.21.15.84:6790/0 conn(0x7f80fc09ac40 legacy=0x7f80fc0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:53.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 1 -- 172.21.15.86:0/2361528006 >> v1:172.21.15.84:6789/0 conn(0x7f80fc0b4360 legacy=0x7f80fc14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:53.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:34:53.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 1 monclient: found mon.noname-c 2024-03-29T10:34:53.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:53.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-29T10:34:53.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 1 -- 172.21.15.86:0/2361528006 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80fc0b5160 con 0x7f80fc09c870 2024-03-29T10:34:53.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 10 monclient: _finish_auth 0 2024-03-29T10:34:53.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:34:53.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:53.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f8101aff640 5 monclient: authenticate success, global_id 4144 2024-03-29T10:34:53.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 1 -- 172.21.15.86:0/2361528006 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f80f0001690 con 0x7f80fc09c870 2024-03-29T10:34:53.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f8101aff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:53.226 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:53.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:53.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 10 monclient: dump: 2024-03-29T10:34:53.227 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:53.227 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:53.227 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.227 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.227 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:53.227 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:53.227 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:53.227 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:53.227 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:53.227 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:53.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f8101aff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:53.227 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 1 -- 172.21.15.86:0/2361528006 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80f00017f0 con 0x7f80fc09c870 2024-03-29T10:34:53.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:53.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f8101aff640 10 monclient: get_monmap_and_config success 2024-03-29T10:34:53.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 1 -- 172.21.15.86:0/2361528006 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f80f0001c00 con 0x7f80fc09c870 2024-03-29T10:34:53.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:53.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f8101aff640 4 set_mon_vals no callback set 2024-03-29T10:34:53.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:53.228 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.230+0000 7f80fa7fc640 10 monclient: dump: 2024-03-29T10:34:53.228 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:53.228 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:53.228 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.228 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.228 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:53.228 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:53.228 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:53.228 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:53.229 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:53.229 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:53.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.231+0000 7f8101aff640 10 monclient: shutdown 2024-03-29T10:34:53.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.231+0000 7f8101aff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:53.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.231+0000 7f8101aff640 1 -- 172.21.15.86:0/2361528006 >> v1:172.21.15.86:6789/0 conn(0x7f80fc09c870 legacy=0x7f80fc13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:53.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.231+0000 7f8101aff640 1 -- 172.21.15.86:0/2361528006 shutdown_connections 2024-03-29T10:34:53.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.231+0000 7f8101aff640 1 -- 172.21.15.86:0/2361528006 >> 172.21.15.86:0/2361528006 conn(0x7f80fc066b00 msgr2=0x7f80fc066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:53.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.231+0000 7f8101aff640 1 -- 172.21.15.86:0/2361528006 shutdown_connections 2024-03-29T10:34:53.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.231+0000 7f8101aff640 1 -- 172.21.15.86:0/2361528006 wait complete. 2024-03-29T10:34:53.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.231+0000 7f8101aff640 10 monclient: build_initial_monmap 2024-03-29T10:34:53.229 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.231+0000 7f8101aff640 10 monclient: monmap: 2024-03-29T10:34:53.229 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-29T10:34:53.229 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:34:53.229 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-03-29T10:34:53.230 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-03-29T10:34:53.230 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:53.230 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:53.230 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:34:53.230 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:34:53.230 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:34:53.230 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:53.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.231+0000 7f8101aff640 1 librados: starting msgr at 2024-03-29T10:34:53.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.231+0000 7f8101aff640 1 librados: starting objecter 2024-03-29T10:34:53.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.232+0000 7f8101aff640 1 Processor -- start 2024-03-29T10:34:53.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.232+0000 7f8101aff640 1 -- start start 2024-03-29T10:34:53.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.232+0000 7f8101aff640 1 librados: setting wanted keys 2024-03-29T10:34:53.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.232+0000 7f8101aff640 1 librados: calling monclient init 2024-03-29T10:34:53.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.232+0000 7f8101aff640 10 monclient: init 2024-03-29T10:34:53.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.232+0000 7f8101aff640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:53.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.232+0000 7f8101aff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:34:53.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.232+0000 7f8101aff640 10 monclient(hunting): picked mon.noname-c con 0x7f80fc09ac40 addr v1:172.21.15.84:6790/0 2024-03-29T10:34:53.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.232+0000 7f8101aff640 10 monclient(hunting): picked mon.noname-a con 0x7f80fc0afa60 addr v1:172.21.15.84:6789/0 2024-03-29T10:34:53.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.232+0000 7f8101aff640 10 monclient(hunting): picked mon.noname-b con 0x7f80fc1db5a0 addr v1:172.21.15.86:6789/0 2024-03-29T10:34:53.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.232+0000 7f8101aff640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80fc0af690 con 0x7f80fc0afa60 2024-03-29T10:34:53.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.232+0000 7f8101aff640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80fc11fbb0 con 0x7f80fc1db5a0 2024-03-29T10:34:53.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.232+0000 7f8101aff640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80fc0afef0 con 0x7f80fc09ac40 2024-03-29T10:34:53.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.232+0000 7f8101aff640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:53.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.232+0000 7f8101aff640 10 monclient(hunting): authenticate will time out at 777.397976s 2024-03-29T10:34:53.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.232+0000 7f80fbfff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f80fc1db5a0 0x7f80fc1e2c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:50368/0 (socket says 172.21.15.86:50368) 2024-03-29T10:34:53.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.232+0000 7f80fb7fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f80fc09ac40 0x7f80fc139490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.86:53926/0 (socket says 172.21.15.86:53926) 2024-03-29T10:34:53.233 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.232+0000 7f80faffd640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f80fc0afa60 0x7f80fc0b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.86:54100/0 (socket says 172.21.15.86:54100) 2024-03-29T10:34:53.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.232+0000 7f80fbfff640 1 -- 172.21.15.86:0/493463546 learned_addr learned my addr 172.21.15.86:0/493463546 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T10:34:53.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 client.?.objecter ms_handle_connect 0x7f80fc1db5a0 2024-03-29T10:34:53.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:53.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 client.?.objecter ms_handle_connect 0x7f80fc09ac40 2024-03-29T10:34:53.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:53.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 client.?.objecter ms_handle_connect 0x7f80fc0afa60 2024-03-29T10:34:53.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:53.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 1 -- 172.21.15.86:0/493463546 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1067022407 0 0) 0x7f80fc0afef0 con 0x7f80fc09ac40 2024-03-29T10:34:53.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:34:53.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:53.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:53.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient(hunting): my global_id is 4151 2024-03-29T10:34:53.234 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 1 -- 172.21.15.86:0/493463546 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80e8004eb0 con 0x7f80fc09ac40 2024-03-29T10:34:53.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 1 -- 172.21.15.86:0/493463546 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1487679761 0 0) 0x7f80fc0af690 con 0x7f80fc0afa60 2024-03-29T10:34:53.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:34:53.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:53.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:53.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient(hunting): my global_id is 4155 2024-03-29T10:34:53.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 1 -- 172.21.15.86:0/493463546 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80fc0afef0 con 0x7f80fc0afa60 2024-03-29T10:34:53.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 1 -- 172.21.15.86:0/493463546 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 191675283 0 0) 0x7f80fc11fbb0 con 0x7f80fc1db5a0 2024-03-29T10:34:53.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:34:53.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:53.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:53.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient(hunting): my global_id is 4147 2024-03-29T10:34:53.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 1 -- 172.21.15.86:0/493463546 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80fc0af690 con 0x7f80fc1db5a0 2024-03-29T10:34:53.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 1 -- 172.21.15.86:0/493463546 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4040600082 0 0) 0x7f80fc0af690 con 0x7f80fc1db5a0 2024-03-29T10:34:53.235 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 1 -- 172.21.15.86:0/493463546 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80fc11fbb0 con 0x7f80fc1db5a0 2024-03-29T10:34:53.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 1 -- 172.21.15.86:0/493463546 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f80f4001900 con 0x7f80fc1db5a0 2024-03-29T10:34:53.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:53.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:34:53.236 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient(hunting): dump: 2024-03-29T10:34:53.236 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:53.236 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:53.236 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.236 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.236 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:53.236 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:53.236 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:53.236 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:53.236 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:53.236 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:53.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:34:53.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 1 -- 172.21.15.86:0/493463546 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1756721367 0 0) 0x7f80fc0afef0 con 0x7f80fc0afa60 2024-03-29T10:34:53.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 1 -- 172.21.15.86:0/493463546 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80fc0af690 con 0x7f80fc0afa60 2024-03-29T10:34:53.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 1 -- 172.21.15.86:0/493463546 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2355207220 0 0) 0x7f80e8004eb0 con 0x7f80fc09ac40 2024-03-29T10:34:53.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 1 -- 172.21.15.86:0/493463546 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80fc0afef0 con 0x7f80fc09ac40 2024-03-29T10:34:53.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 1 -- 172.21.15.86:0/493463546 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f80f0001b00 con 0x7f80fc0afa60 2024-03-29T10:34:53.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:53.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:34:53.237 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient(hunting): dump: 2024-03-29T10:34:53.237 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:53.237 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:53.237 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.237 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.237 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:53.238 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:53.238 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:53.238 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:53.238 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:53.238 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:53.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 1 -- 172.21.15.86:0/493463546 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f80ec001970 con 0x7f80fc09ac40 2024-03-29T10:34:53.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:53.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:34:53.238 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient(hunting): dump: 2024-03-29T10:34:53.238 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:53.238 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:53.238 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.238 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.239 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:53.239 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:53.239 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:53.239 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:53.239 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:53.239 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:53.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 1 -- 172.21.15.86:0/493463546 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2729788934 0 0) 0x7f80fc11fbb0 con 0x7f80fc1db5a0 2024-03-29T10:34:53.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 1 -- 172.21.15.86:0/493463546 >> v1:172.21.15.84:6790/0 conn(0x7f80fc09ac40 legacy=0x7f80fc139490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:53.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 1 -- 172.21.15.86:0/493463546 >> v1:172.21.15.84:6789/0 conn(0x7f80fc0afa60 legacy=0x7f80fc0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:53.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:34:53.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 1 monclient: found mon.b 2024-03-29T10:34:53.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:53.239 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:53.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 1 -- 172.21.15.86:0/493463546 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80fc1dab60 con 0x7f80fc1db5a0 2024-03-29T10:34:53.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient: _finish_auth 0 2024-03-29T10:34:53.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:34:53.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:53.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f8101aff640 5 monclient: authenticate success, global_id 4147 2024-03-29T10:34:53.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f8101aff640 10 monclient: _renew_subs 2024-03-29T10:34:53.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f8101aff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:53.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f8101aff640 1 -- 172.21.15.86:0/493463546 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80fc1dadd0 con 0x7f80fc1db5a0 2024-03-29T10:34:53.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f8101aff640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:34:53.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f8101aff640 10 monclient: _renew_subs 2024-03-29T10:34:53.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f8101aff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:53.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f8101aff640 1 -- 172.21.15.86:0/493463546 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80fc139d80 con 0x7f80fc1db5a0 2024-03-29T10:34:53.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f8101aff640 1 librados: init done 2024-03-29T10:34:53.240 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 1 -- 172.21.15.86:0/493463546 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80f4001e70 con 0x7f80fc1db5a0 2024-03-29T10:34:53.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:53.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 1 -- 172.21.15.86:0/493463546 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f80f4003b30 con 0x7f80fc1db5a0 2024-03-29T10:34:53.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:53.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:53.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80f9ffb640 4 set_mon_vals no callback set 2024-03-29T10:34:53.241 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 monclient: dump: 2024-03-29T10:34:53.241 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:53.241 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:53.241 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.241 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.241 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:53.241 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:53.241 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:53.241 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:53.242 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:53.242 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:53.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 1 -- 172.21.15.86:0/493463546 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f80f40040a0 con 0x7f80fc1db5a0 2024-03-29T10:34:53.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 1 -- 172.21.15.86:0/493463546 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 659743224 0 0) 0x7f80f40068a0 con 0x7f80fc1db5a0 2024-03-29T10:34:53.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 10 client.4147.objecter ms_dispatch 0x7f80fc000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T10:34:53.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 3 client.4147.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T10:34:53.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 3 client.4147.objecter handle_osd_map decoding full epoch 8 2024-03-29T10:34:53.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.233+0000 7f80e3fff640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-29T10:34:53.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.234+0000 7f8101aff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:53.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.234+0000 7f8101aff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:53.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.234+0000 7f8101aff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:53.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.234+0000 7f8101aff640 1 -- 172.21.15.86:0/493463546 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80dc005080 con 0x7f80fc1db5a0 2024-03-29T10:34:53.242 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.236+0000 7f80e3fff640 1 -- 172.21.15.86:0/493463546 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f80f4005b40 con 0x7f80fc1db5a0 2024-03-29T10:34:53.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.236+0000 7f80e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:53.243 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.236+0000 7f80e3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:34:53.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.307+0000 7f8101aff640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a6752d73-9b2c-451f-92fe-1baa4b29cc9a", "id": 6}] 2024-03-29T10:34:53.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.307+0000 7f8101aff640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a6752d73-9b2c-451f-92fe-1baa4b29cc9a", "id": 6}] 2024-03-29T10:34:53.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.307+0000 7f8101aff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:53.293 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.307+0000 7f8101aff640 1 -- 172.21.15.86:0/493463546 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd new", "uuid": "a6752d73-9b2c-451f-92fe-1baa4b29cc9a", "id": 6} v 0) v1 -- 0x7f80dc0058b0 con 0x7f80fc1db5a0 2024-03-29T10:34:53.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.311+0000 7f80e3fff640 1 -- 172.21.15.86:0/493463546 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a6752d73-9b2c-451f-92fe-1baa4b29cc9a", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 4175193934 0 1113811849) 0x7f80f4005d60 con 0x7f80fc1db5a0 2024-03-29T10:34:53.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.311+0000 7f80e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a6752d73-9b2c-451f-92fe-1baa4b29cc9a", "id": 6}] 2024-03-29T10:34:53.296 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.311+0000 7f80e3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:34:53.297 INFO:teuthology.orchestra.run.smithi086.stdout:6 2024-03-29T10:34:53.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.312+0000 7f8101aff640 10 librados: watch_flush enter 2024-03-29T10:34:53.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.312+0000 7f8101aff640 10 librados: watch_flush exit 2024-03-29T10:34:53.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.312+0000 7f8101aff640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-29T10:34:53.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.312+0000 7f8101aff640 10 client.4147.objecter successfully canceled tick 2024-03-29T10:34:53.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.312+0000 7f8101aff640 10 monclient: shutdown 2024-03-29T10:34:53.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.312+0000 7f8101aff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:53.298 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.312+0000 7f8101aff640 1 -- 172.21.15.86:0/493463546 >> v1:172.21.15.86:6789/0 conn(0x7f80fc1db5a0 legacy=0x7f80fc1e2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:53.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.312+0000 7f8101aff640 1 -- 172.21.15.86:0/493463546 shutdown_connections 2024-03-29T10:34:53.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.312+0000 7f8101aff640 1 -- 172.21.15.86:0/493463546 >> 172.21.15.86:0/493463546 conn(0x7f80fc066b00 msgr2=0x7f80fc0b7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:53.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.312+0000 7f8101aff640 1 -- 172.21.15.86:0/493463546 shutdown_connections 2024-03-29T10:34:53.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.312+0000 7f8101aff640 1 -- 172.21.15.86:0/493463546 wait complete. 2024-03-29T10:34:53.299 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.312+0000 7f8101aff640 1 librados: shutdown 2024-03-29T10:34:53.308 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd new 4d1510b3-682c-4fe9-a014-152f64ddbd37 7 2024-03-29T10:34:53.336 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T10:34:53.350+0000 7f187bc42040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:34:53.368 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-29T10:34:53.383+0000 7f4e58c21040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:34:53.409 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.424+0000 7f988c37b640 10 monclient: get_monmap_and_config 2024-03-29T10:34:53.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.424+0000 7f988c37b640 10 monclient: build_initial_monmap 2024-03-29T10:34:53.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.424+0000 7f988c37b640 10 monclient: monmap: 2024-03-29T10:34:53.410 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-29T10:34:53.410 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:53.410 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:53.425237+0000 2024-03-29T10:34:53.410 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:53.425237+0000 2024-03-29T10:34:53.410 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:53.410 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:53.410 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:34:53.410 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:34:53.410 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:34:53.410 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:53.410 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.424+0000 7f988c37b640 1 Processor -- start 2024-03-29T10:34:53.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.424+0000 7f988c37b640 1 -- start start 2024-03-29T10:34:53.411 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.424+0000 7f988c37b640 10 monclient: init 2024-03-29T10:34:53.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.425+0000 7f988c37b640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:53.418 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.425+0000 7f988c37b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:34:53.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.425+0000 7f988c37b640 10 monclient(hunting): picked mon.noname-a con 0x7f988409ac40 addr v1:172.21.15.84:6789/0 2024-03-29T10:34:53.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.425+0000 7f988c37b640 10 monclient(hunting): picked mon.noname-b con 0x7f988409c870 addr v1:172.21.15.84:6790/0 2024-03-29T10:34:53.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.425+0000 7f988c37b640 10 monclient(hunting): picked mon.noname-c con 0x7f98840b4360 addr v1:172.21.15.86:6789/0 2024-03-29T10:34:53.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.425+0000 7f988c37b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988409dec0 con 0x7f988409ac40 2024-03-29T10:34:53.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.425+0000 7f988c37b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988409da30 con 0x7f98840b4360 2024-03-29T10:34:53.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.425+0000 7f988c37b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98840b39c0 con 0x7f988409c870 2024-03-29T10:34:53.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.425+0000 7f988c37b640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:53.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.425+0000 7f988c37b640 10 monclient(hunting): authenticate will time out at 777.590614s 2024-03-29T10:34:53.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.425+0000 7f988a918640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f98840b4360 0x7f988414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:50384/0 (socket says 172.21.15.86:50384) 2024-03-29T10:34:53.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.425+0000 7f988a918640 1 -- 172.21.15.86:0/1754910989 learned_addr learned my addr 172.21.15.86:0/1754910989 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T10:34:53.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.425+0000 7f9889115640 1 -- 172.21.15.86:0/1754910989 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 903016968 0 0) 0x7f988409da30 con 0x7f98840b4360 2024-03-29T10:34:53.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.425+0000 7f9889115640 10 monclient(hunting): _negotiate 2024-03-29T10:34:53.419 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.425+0000 7f9889115640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:53.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.425+0000 7f9889115640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:53.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.425+0000 7f9889115640 10 monclient(hunting): my global_id is 4150 2024-03-29T10:34:53.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.425+0000 7f9889115640 1 -- 172.21.15.86:0/1754910989 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9870003520 con 0x7f98840b4360 2024-03-29T10:34:53.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.425+0000 7f9889115640 1 -- 172.21.15.86:0/1754910989 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1016124808 0 0) 0x7f9880001110 con 0x7f988409ac40 2024-03-29T10:34:53.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 10 monclient(hunting): _negotiate 2024-03-29T10:34:53.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:53.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:53.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 10 monclient(hunting): my global_id is 4158 2024-03-29T10:34:53.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 1 -- 172.21.15.86:0/1754910989 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f988409da30 con 0x7f988409ac40 2024-03-29T10:34:53.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 1 -- 172.21.15.86:0/1754910989 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848423520 0 0) 0x7f98840b39c0 con 0x7f988409c870 2024-03-29T10:34:53.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 10 monclient(hunting): _negotiate 2024-03-29T10:34:53.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:53.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:53.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 10 monclient(hunting): my global_id is 4154 2024-03-29T10:34:53.420 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 1 -- 172.21.15.86:0/1754910989 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9880001110 con 0x7f988409c870 2024-03-29T10:34:53.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 1 -- 172.21.15.86:0/1754910989 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3449863330 0 0) 0x7f9870003520 con 0x7f98840b4360 2024-03-29T10:34:53.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 1 -- 172.21.15.86:0/1754910989 >> v1:172.21.15.84:6790/0 conn(0x7f988409c870 legacy=0x7f988413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:53.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 1 -- 172.21.15.86:0/1754910989 >> v1:172.21.15.84:6789/0 conn(0x7f988409ac40 legacy=0x7f98840b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:53.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 10 monclient: _finish_hunting 0 2024-03-29T10:34:53.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 1 monclient: found mon.noname-c 2024-03-29T10:34:53.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:53.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-29T10:34:53.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 1 -- 172.21.15.86:0/1754910989 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98840b5160 con 0x7f98840b4360 2024-03-29T10:34:53.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 10 monclient: _finish_auth 0 2024-03-29T10:34:53.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 10 monclient: _check_auth_tickets 2024-03-29T10:34:53.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:53.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 1 -- 172.21.15.86:0/1754910989 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f987c001a10 con 0x7f98840b4360 2024-03-29T10:34:53.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f988c37b640 5 monclient: authenticate success, global_id 4150 2024-03-29T10:34:53.421 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f988c37b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:53.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:53.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:53.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 10 monclient: dump: 2024-03-29T10:34:53.422 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:53.422 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:53.422 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.422 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.422 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:53.422 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:53.422 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:53.422 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:53.422 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:53.422 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:53.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f988c37b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:53.422 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 1 -- 172.21.15.86:0/1754910989 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f987c001bf0 con 0x7f98840b4360 2024-03-29T10:34:53.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:53.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 1 -- 172.21.15.86:0/1754910989 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f987c002000 con 0x7f98840b4360 2024-03-29T10:34:53.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f988c37b640 10 monclient: get_monmap_and_config success 2024-03-29T10:34:53.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f988c37b640 4 set_mon_vals no callback set 2024-03-29T10:34:53.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:53.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:53.423 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f9889115640 10 monclient: dump: 2024-03-29T10:34:53.423 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:53.423 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:53.423 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.423 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.423 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:53.423 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:53.423 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:53.424 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:53.424 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:53.424 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:53.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f988c37b640 10 monclient: shutdown 2024-03-29T10:34:53.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f988c37b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:53.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f988c37b640 1 -- 172.21.15.86:0/1754910989 >> v1:172.21.15.86:6789/0 conn(0x7f98840b4360 legacy=0x7f988414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:53.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f988c37b640 1 -- 172.21.15.86:0/1754910989 shutdown_connections 2024-03-29T10:34:53.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f988c37b640 1 -- 172.21.15.86:0/1754910989 >> 172.21.15.86:0/1754910989 conn(0x7f9884066b00 msgr2=0x7f9884066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:53.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f988c37b640 1 -- 172.21.15.86:0/1754910989 shutdown_connections 2024-03-29T10:34:53.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.426+0000 7f988c37b640 1 -- 172.21.15.86:0/1754910989 wait complete. 2024-03-29T10:34:53.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.427+0000 7f988c37b640 10 monclient: build_initial_monmap 2024-03-29T10:34:53.424 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.427+0000 7f988c37b640 10 monclient: monmap: 2024-03-29T10:34:53.424 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-29T10:34:53.424 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:34:53.425 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-03-29T10:34:53.425 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-03-29T10:34:53.425 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:53.425 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:53.425 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:34:53.425 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:34:53.425 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:34:53.425 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:53.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.427+0000 7f988c37b640 1 librados: starting msgr at 2024-03-29T10:34:53.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.427+0000 7f988c37b640 1 librados: starting objecter 2024-03-29T10:34:53.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.427+0000 7f988c37b640 1 Processor -- start 2024-03-29T10:34:53.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.427+0000 7f988c37b640 1 -- start start 2024-03-29T10:34:53.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.427+0000 7f988c37b640 1 librados: setting wanted keys 2024-03-29T10:34:53.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.427+0000 7f988c37b640 1 librados: calling monclient init 2024-03-29T10:34:53.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.427+0000 7f988c37b640 10 monclient: init 2024-03-29T10:34:53.425 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f988c37b640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:53.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f988c37b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:34:53.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f988c37b640 10 monclient(hunting): picked mon.noname-b con 0x7f988409ac40 addr v1:172.21.15.86:6789/0 2024-03-29T10:34:53.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f988c37b640 10 monclient(hunting): picked mon.noname-a con 0x7f9884141910 addr v1:172.21.15.84:6789/0 2024-03-29T10:34:53.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f988c37b640 10 monclient(hunting): picked mon.noname-c con 0x7f988413ffa0 addr v1:172.21.15.84:6790/0 2024-03-29T10:34:53.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f988c37b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9884139b60 con 0x7f9884141910 2024-03-29T10:34:53.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f988c37b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98841d6f10 con 0x7f988409ac40 2024-03-29T10:34:53.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f988c37b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98840afba0 con 0x7f988413ffa0 2024-03-29T10:34:53.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f988c37b640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:53.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f988c37b640 10 monclient(hunting): authenticate will time out at 777.593565s 2024-03-29T10:34:53.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f988a117640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f988409ac40 0x7f9884141200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:50388/0 (socket says 172.21.15.86:50388) 2024-03-29T10:34:53.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f988a117640 1 -- 172.21.15.86:0/4198955170 learned_addr learned my addr 172.21.15.86:0/4198955170 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T10:34:53.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 10 client.?.objecter ms_handle_connect 0x7f988409ac40 2024-03-29T10:34:53.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:53.426 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 10 client.?.objecter ms_handle_connect 0x7f9884141910 2024-03-29T10:34:53.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:53.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 10 client.?.objecter ms_handle_connect 0x7f988413ffa0 2024-03-29T10:34:53.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:53.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 1 -- 172.21.15.86:0/4198955170 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518690471 0 0) 0x7f98841d6f10 con 0x7f988409ac40 2024-03-29T10:34:53.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:53.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:53.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:53.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 10 monclient(hunting): my global_id is 4153 2024-03-29T10:34:53.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 1 -- 172.21.15.86:0/4198955170 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9868004eb0 con 0x7f988409ac40 2024-03-29T10:34:53.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 1 -- 172.21.15.86:0/4198955170 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 393238188 0 0) 0x7f9880001110 con 0x7f9884141910 2024-03-29T10:34:53.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:53.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:53.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:53.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 10 monclient(hunting): my global_id is 4161 2024-03-29T10:34:53.427 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 1 -- 172.21.15.86:0/4198955170 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98841d6f10 con 0x7f9884141910 2024-03-29T10:34:53.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 1 -- 172.21.15.86:0/4198955170 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 946742159 0 0) 0x7f98840afba0 con 0x7f988413ffa0 2024-03-29T10:34:53.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 10 monclient(hunting): _negotiate 2024-03-29T10:34:53.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:53.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:53.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 10 monclient(hunting): my global_id is 4157 2024-03-29T10:34:53.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 1 -- 172.21.15.86:0/4198955170 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9880001110 con 0x7f988413ffa0 2024-03-29T10:34:53.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 1 -- 172.21.15.86:0/4198955170 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3082817990 0 0) 0x7f9868004eb0 con 0x7f988409ac40 2024-03-29T10:34:53.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 1 -- 172.21.15.86:0/4198955170 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98840afba0 con 0x7f988409ac40 2024-03-29T10:34:53.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 1 -- 172.21.15.86:0/4198955170 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f98800019a0 con 0x7f988409ac40 2024-03-29T10:34:53.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:53.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:34:53.428 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 10 monclient(hunting): dump: 2024-03-29T10:34:53.428 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:53.428 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:53.428 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.429 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.429 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:53.429 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:53.429 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:53.429 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:53.429 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:53.429 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:53.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.428+0000 7f987affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:34:53.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 1 -- 172.21.15.86:0/4198955170 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2678633784 0 0) 0x7f9880001110 con 0x7f988413ffa0 2024-03-29T10:34:53.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 1 -- 172.21.15.86:0/4198955170 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9868004eb0 con 0x7f988413ffa0 2024-03-29T10:34:53.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 1 -- 172.21.15.86:0/4198955170 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2365543462 0 0) 0x7f98841d6f10 con 0x7f9884141910 2024-03-29T10:34:53.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 1 -- 172.21.15.86:0/4198955170 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9880001110 con 0x7f9884141910 2024-03-29T10:34:53.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 1 -- 172.21.15.86:0/4198955170 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f987c001f00 con 0x7f988413ffa0 2024-03-29T10:34:53.429 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:53.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:34:53.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 10 monclient(hunting): dump: 2024-03-29T10:34:53.430 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:53.430 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:53.430 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.430 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.430 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:53.430 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:53.430 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:53.430 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:53.430 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:53.430 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:53.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 1 -- 172.21.15.86:0/4198955170 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3259931753 0 0) 0x7f98840afba0 con 0x7f988409ac40 2024-03-29T10:34:53.430 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 1 -- 172.21.15.86:0/4198955170 >> v1:172.21.15.84:6790/0 conn(0x7f988413ffa0 legacy=0x7f9884138e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:53.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 1 -- 172.21.15.86:0/4198955170 >> v1:172.21.15.84:6789/0 conn(0x7f9884141910 legacy=0x7f98841d73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:53.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 10 monclient: _finish_hunting 0 2024-03-29T10:34:53.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 1 monclient: found mon.b 2024-03-29T10:34:53.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:53.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:53.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 1 -- 172.21.15.86:0/4198955170 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9884139520 con 0x7f988409ac40 2024-03-29T10:34:53.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 10 monclient: _finish_auth 0 2024-03-29T10:34:53.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 10 monclient: _check_auth_tickets 2024-03-29T10:34:53.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:53.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f988c37b640 5 monclient: authenticate success, global_id 4153 2024-03-29T10:34:53.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f988c37b640 10 monclient: _renew_subs 2024-03-29T10:34:53.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f988c37b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:53.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f988c37b640 1 -- 172.21.15.86:0/4198955170 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9884139710 con 0x7f988409ac40 2024-03-29T10:34:53.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f988c37b640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:34:53.431 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f988c37b640 10 monclient: _renew_subs 2024-03-29T10:34:53.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f988c37b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:53.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f988c37b640 1 -- 172.21.15.86:0/4198955170 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98841d7c90 con 0x7f988409ac40 2024-03-29T10:34:53.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f988c37b640 1 librados: init done 2024-03-29T10:34:53.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 1 -- 172.21.15.86:0/4198955170 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9880001c40 con 0x7f988409ac40 2024-03-29T10:34:53.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:53.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 1 -- 172.21.15.86:0/4198955170 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f9880003a60 con 0x7f988409ac40 2024-03-29T10:34:53.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:53.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:53.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f9888914640 4 set_mon_vals no callback set 2024-03-29T10:34:53.432 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 10 monclient: dump: 2024-03-29T10:34:53.432 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T10:34:53.432 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:53.432 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.432 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.432 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:53.432 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T10:34:53.433 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:53.433 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:53.433 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:53.433 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:34:53.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 1 -- 172.21.15.86:0/4198955170 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9880004010 con 0x7f988409ac40 2024-03-29T10:34:53.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 1 -- 172.21.15.86:0/4198955170 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 641017364 0 0) 0x7f9880006900 con 0x7f988409ac40 2024-03-29T10:34:53.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 10 client.4153.objecter ms_dispatch 0x7f9884000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T10:34:53.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 3 client.4153.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T10:34:53.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 3 client.4153.objecter handle_osd_map decoding full epoch 9 2024-03-29T10:34:53.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.429+0000 7f987affd640 20 client.4153.objecter dump_active .. 0 homeless 2024-03-29T10:34:53.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.430+0000 7f988c37b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:53.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.430+0000 7f988c37b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:53.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.430+0000 7f988c37b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:53.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.430+0000 7f988c37b640 1 -- 172.21.15.86:0/4198955170 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f986c005080 con 0x7f988409ac40 2024-03-29T10:34:53.433 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.431+0000 7f987affd640 1 -- 172.21.15.86:0/4198955170 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9880004450 con 0x7f988409ac40 2024-03-29T10:34:53.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.431+0000 7f987affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:53.434 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.431+0000 7f987affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:34:53.472 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T10:34:53.486+0000 7f187bc42040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:34:53.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.503+0000 7f988c37b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4d1510b3-682c-4fe9-a014-152f64ddbd37", "id": 7}] 2024-03-29T10:34:53.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.503+0000 7f988c37b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4d1510b3-682c-4fe9-a014-152f64ddbd37", "id": 7}] 2024-03-29T10:34:53.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.503+0000 7f988c37b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:34:53.489 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.503+0000 7f988c37b640 1 -- 172.21.15.86:0/4198955170 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd new", "uuid": "4d1510b3-682c-4fe9-a014-152f64ddbd37", "id": 7} v 0) v1 -- 0x7f986c0058b0 con 0x7f988409ac40 2024-03-29T10:34:53.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.507+0000 7f987affd640 1 -- 172.21.15.86:0/4198955170 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4d1510b3-682c-4fe9-a014-152f64ddbd37", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 298802993 0 1371665406) 0x7f9880006b90 con 0x7f988409ac40 2024-03-29T10:34:53.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.507+0000 7f987affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4d1510b3-682c-4fe9-a014-152f64ddbd37", "id": 7}] 2024-03-29T10:34:53.492 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.507+0000 7f987affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:34:53.492 INFO:teuthology.orchestra.run.smithi086.stdout:7 2024-03-29T10:34:53.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.507+0000 7f988c37b640 10 librados: watch_flush enter 2024-03-29T10:34:53.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.507+0000 7f988c37b640 10 librados: watch_flush exit 2024-03-29T10:34:53.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.508+0000 7f988c37b640 20 client.4153.objecter shutdown clearing up homeless session... 2024-03-29T10:34:53.493 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.508+0000 7f988c37b640 10 client.4153.objecter successfully canceled tick 2024-03-29T10:34:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.508+0000 7f988c37b640 10 monclient: shutdown 2024-03-29T10:34:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.508+0000 7f988c37b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.508+0000 7f988c37b640 1 -- 172.21.15.86:0/4198955170 >> v1:172.21.15.86:6789/0 conn(0x7f988409ac40 legacy=0x7f9884141200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.508+0000 7f988c37b640 1 -- 172.21.15.86:0/4198955170 shutdown_connections 2024-03-29T10:34:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.508+0000 7f988c37b640 1 -- 172.21.15.86:0/4198955170 >> 172.21.15.86:0/4198955170 conn(0x7f9884066b00 msgr2=0x7f988413c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.508+0000 7f988c37b640 1 -- 172.21.15.86:0/4198955170 shutdown_connections 2024-03-29T10:34:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.508+0000 7f988c37b640 1 -- 172.21.15.86:0/4198955170 wait complete. 2024-03-29T10:34:53.494 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T10:34:53.508+0000 7f988c37b640 1 librados: shutdown 2024-03-29T10:34:53.503 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T10:34:53.503 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T10:34:53.506 INFO:tasks.ceph.osd.0:Started 2024-03-29T10:34:53.506 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T10:34:53.506 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T10:34:53.508 INFO:tasks.ceph.osd.1:Started 2024-03-29T10:34:53.508 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T10:34:53.508 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T10:34:53.510 INFO:tasks.ceph.osd.2:Started 2024-03-29T10:34:53.510 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T10:34:53.510 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T10:34:53.511 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-29T10:34:53.525+0000 7f4e58c21040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:34:53.512 INFO:tasks.ceph.osd.3:Started 2024-03-29T10:34:53.513 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T10:34:53.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T10:34:53.515 INFO:tasks.ceph.osd.4:Started 2024-03-29T10:34:53.515 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T10:34:53.515 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T10:34:53.517 INFO:tasks.ceph.osd.5:Started 2024-03-29T10:34:53.517 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T10:34:53.517 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T10:34:53.519 INFO:tasks.ceph.osd.6:Started 2024-03-29T10:34:53.519 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T10:34:53.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T10:34:53.521 INFO:tasks.ceph.osd.7:Started 2024-03-29T10:34:53.522 DEBUG:teuthology.orchestra.run.smithi084:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:34:53.635 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T10:34:53.648+0000 7f187bc42040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:34:53.637 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.650+0000 7fe86fbd6640 10 monclient: get_monmap_and_config 2024-03-29T10:34:53.637 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.651+0000 7fe86fbd6640 10 monclient: build_initial_monmap 2024-03-29T10:34:53.637 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.651+0000 7fe86fbd6640 10 monclient: monmap: 2024-03-29T10:34:53.637 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 0 2024-03-29T10:34:53.637 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:53.637 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-29T10:34:53.652959+0000 2024-03-29T10:34:53.637 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-29T10:34:53.652959+0000 2024-03-29T10:34:53.637 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:53.637 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 1 2024-03-29T10:34:53.637 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:34:53.637 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:34:53.637 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:34:53.637 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T10:34:53.646 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.651+0000 7fe86fbd6640 1 Processor -- start 2024-03-29T10:34:53.646 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.651+0000 7fe86fbd6640 1 -- start start 2024-03-29T10:34:53.646 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.651+0000 7fe86fbd6640 10 monclient: init 2024-03-29T10:34:53.646 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.652+0000 7fe86fbd6640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:53.646 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.652+0000 7fe86fbd6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:34:53.646 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.652+0000 7fe86fbd6640 10 monclient(hunting): picked mon.noname-b con 0x7fe86809ac40 addr v1:172.21.15.84:6790/0 2024-03-29T10:34:53.646 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.652+0000 7fe86fbd6640 10 monclient(hunting): picked mon.noname-c con 0x7fe86809c870 addr v1:172.21.15.86:6789/0 2024-03-29T10:34:53.646 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.652+0000 7fe86fbd6640 10 monclient(hunting): picked mon.noname-a con 0x7fe8680b4360 addr v1:172.21.15.84:6789/0 2024-03-29T10:34:53.647 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.652+0000 7fe86fbd6640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe86809dec0 con 0x7fe8680b4360 2024-03-29T10:34:53.647 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.652+0000 7fe86fbd6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe86809da30 con 0x7fe86809c870 2024-03-29T10:34:53.647 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.652+0000 7fe86fbd6640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8680b39c0 con 0x7fe86809ac40 2024-03-29T10:34:53.647 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.652+0000 7fe86fbd6640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:53.647 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.652+0000 7fe86e173640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fe8680b4360 0x7fe86814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:56220/0 (socket says 172.21.15.84:56220) 2024-03-29T10:34:53.647 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.652+0000 7fe86e173640 1 -- 172.21.15.84:0/2136893790 learned_addr learned my addr 172.21.15.84:0/2136893790 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:34:53.647 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.652+0000 7fe86fbd6640 10 monclient(hunting): authenticate will time out at 795.916663s 2024-03-29T10:34:53.647 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.652+0000 7fe86d972640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fe86809ac40 0x7fe8680b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:51012/0 (socket says 172.21.15.84:51012) 2024-03-29T10:34:53.647 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.652+0000 7fe86c970640 1 -- 172.21.15.84:0/2136893790 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1522817383 0 0) 0x7fe860001490 con 0x7fe8680b4360 2024-03-29T10:34:53.647 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.652+0000 7fe86c970640 10 monclient(hunting): _negotiate 2024-03-29T10:34:53.647 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.652+0000 7fe86c970640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:53.647 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.652+0000 7fe86c970640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:53.647 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.652+0000 7fe86c970640 10 monclient(hunting): my global_id is 4188 2024-03-29T10:34:53.647 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.652+0000 7fe86c970640 1 -- 172.21.15.84:0/2136893790 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe854003520 con 0x7fe8680b4360 2024-03-29T10:34:53.648 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.652+0000 7fe86c970640 1 -- 172.21.15.84:0/2136893790 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486118467 0 0) 0x7fe8680b39c0 con 0x7fe86809ac40 2024-03-29T10:34:53.648 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.652+0000 7fe86c970640 10 monclient(hunting): _negotiate 2024-03-29T10:34:53.648 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.652+0000 7fe86c970640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:53.648 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.652+0000 7fe86c970640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:53.648 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.652+0000 7fe86c970640 10 monclient(hunting): my global_id is 4184 2024-03-29T10:34:53.648 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.652+0000 7fe86c970640 1 -- 172.21.15.84:0/2136893790 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe860001490 con 0x7fe86809ac40 2024-03-29T10:34:53.648 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86c970640 1 -- 172.21.15.84:0/2136893790 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3377249239 0 0) 0x7fe860001490 con 0x7fe86809ac40 2024-03-29T10:34:53.648 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86c970640 1 -- 172.21.15.84:0/2136893790 >> v1:172.21.15.86:6789/0 conn(0x7fe86809c870 legacy=0x7fe86813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:53.648 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86c970640 1 -- 172.21.15.84:0/2136893790 >> v1:172.21.15.84:6789/0 conn(0x7fe8680b4360 legacy=0x7fe86814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:53.648 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86c970640 10 monclient: _finish_hunting 0 2024-03-29T10:34:53.648 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86c970640 1 monclient: found mon.noname-b 2024-03-29T10:34:53.648 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86c970640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:53.648 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86c970640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:34:53.649 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86c970640 1 -- 172.21.15.84:0/2136893790 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8680b5160 con 0x7fe86809ac40 2024-03-29T10:34:53.649 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86c970640 10 monclient: _finish_auth 0 2024-03-29T10:34:53.649 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86c970640 10 monclient: _check_auth_tickets 2024-03-29T10:34:53.649 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86c970640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:53.649 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86fbd6640 5 monclient: authenticate success, global_id 4184 2024-03-29T10:34:53.649 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86c970640 1 -- 172.21.15.84:0/2136893790 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fe858001670 con 0x7fe86809ac40 2024-03-29T10:34:53.649 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86fbd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:53.649 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86c970640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:53.649 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86c970640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:34:53.649 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86c970640 10 monclient: dump: 2024-03-29T10:34:53.649 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 1 2024-03-29T10:34:53.649 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:53.649 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.649 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.650 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:53.650 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 1 2024-03-29T10:34:53.650 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:53.650 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:53.650 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:53.650 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T10:34:53.650 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86fbd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:53.650 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86c970640 1 -- 172.21.15.84:0/2136893790 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8580017d0 con 0x7fe86809ac40 2024-03-29T10:34:53.650 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86c970640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:53.650 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86fbd6640 10 monclient: get_monmap_and_config success 2024-03-29T10:34:53.650 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86c970640 1 -- 172.21.15.84:0/2136893790 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fe858001c20 con 0x7fe86809ac40 2024-03-29T10:34:53.650 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86c970640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:53.650 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86fbd6640 4 set_mon_vals no callback set 2024-03-29T10:34:53.651 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86c970640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:34:53.651 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86c970640 10 monclient: dump: 2024-03-29T10:34:53.651 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 1 2024-03-29T10:34:53.651 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:53.651 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.651 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.651 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:53.651 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 1 2024-03-29T10:34:53.651 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:53.651 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:53.651 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:53.651 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T10:34:53.651 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86fbd6640 10 monclient: shutdown 2024-03-29T10:34:53.651 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86fbd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:53.651 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86fbd6640 1 -- 172.21.15.84:0/2136893790 >> v1:172.21.15.84:6790/0 conn(0x7fe86809ac40 legacy=0x7fe8680b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:53.652 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86fbd6640 1 -- 172.21.15.84:0/2136893790 shutdown_connections 2024-03-29T10:34:53.652 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86fbd6640 1 -- 172.21.15.84:0/2136893790 >> 172.21.15.84:0/2136893790 conn(0x7fe868066b00 msgr2=0x7fe868066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:53.652 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86fbd6640 1 -- 172.21.15.84:0/2136893790 shutdown_connections 2024-03-29T10:34:53.652 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.653+0000 7fe86fbd6640 1 -- 172.21.15.84:0/2136893790 wait complete. 2024-03-29T10:34:53.652 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.654+0000 7fe86fbd6640 10 monclient: build_initial_monmap 2024-03-29T10:34:53.652 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.654+0000 7fe86fbd6640 10 monclient: monmap: 2024-03-29T10:34:53.652 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 0 2024-03-29T10:34:53.652 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:34:53.652 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 0.000000 2024-03-29T10:34:53.652 INFO:teuthology.orchestra.run.smithi084.stderr:created 0.000000 2024-03-29T10:34:53.652 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:53.652 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 1 2024-03-29T10:34:53.652 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:34:53.653 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:34:53.653 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:34:53.653 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T10:34:53.653 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.654+0000 7fe86fbd6640 1 librados: starting msgr at 2024-03-29T10:34:53.653 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.654+0000 7fe86fbd6640 1 librados: starting objecter 2024-03-29T10:34:53.653 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.654+0000 7fe86fbd6640 1 Processor -- start 2024-03-29T10:34:53.653 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.654+0000 7fe86fbd6640 1 -- start start 2024-03-29T10:34:53.653 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.654+0000 7fe86fbd6640 1 librados: setting wanted keys 2024-03-29T10:34:53.653 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.654+0000 7fe86fbd6640 1 librados: calling monclient init 2024-03-29T10:34:53.653 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.654+0000 7fe86fbd6640 10 monclient: init 2024-03-29T10:34:53.653 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe86fbd6640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:53.653 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe86fbd6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:34:53.653 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe86fbd6640 10 monclient(hunting): picked mon.noname-a con 0x7fe86809ac40 addr v1:172.21.15.84:6789/0 2024-03-29T10:34:53.653 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe86fbd6640 10 monclient(hunting): picked mon.noname-b con 0x7fe8681da7a0 addr v1:172.21.15.86:6789/0 2024-03-29T10:34:53.653 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe86fbd6640 10 monclient(hunting): picked mon.noname-c con 0x7fe868146130 addr v1:172.21.15.84:6790/0 2024-03-29T10:34:53.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe86fbd6640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe868142a60 con 0x7fe86809ac40 2024-03-29T10:34:53.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe86fbd6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe868141390 con 0x7fe8681da7a0 2024-03-29T10:34:53.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe86fbd6640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8681418b0 con 0x7fe868146130 2024-03-29T10:34:53.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe86fbd6640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:53.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe86fbd6640 10 monclient(hunting): authenticate will time out at 795.919448s 2024-03-29T10:34:53.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe86d972640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fe86809ac40 0x7fe8681da090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:56232/0 (socket says 172.21.15.84:56232) 2024-03-29T10:34:53.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe86d972640 1 -- 172.21.15.84:0/382907140 learned_addr learned my addr 172.21.15.84:0/382907140 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:34:53.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 10 client.?.objecter ms_handle_connect 0x7fe86809ac40 2024-03-29T10:34:53.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:53.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 10 client.?.objecter ms_handle_connect 0x7fe868146130 2024-03-29T10:34:53.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:53.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 1 -- 172.21.15.84:0/382907140 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722147257 0 0) 0x7fe868142a60 con 0x7fe86809ac40 2024-03-29T10:34:53.654 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:34:53.655 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:53.655 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:53.655 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 10 monclient(hunting): my global_id is 4191 2024-03-29T10:34:53.655 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 1 -- 172.21.15.84:0/382907140 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe84c004eb0 con 0x7fe86809ac40 2024-03-29T10:34:53.655 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 1 -- 172.21.15.84:0/382907140 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1236049153 0 0) 0x7fe8681418b0 con 0x7fe868146130 2024-03-29T10:34:53.655 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:34:53.655 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:53.655 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:53.655 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 10 monclient(hunting): my global_id is 4187 2024-03-29T10:34:53.655 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 1 -- 172.21.15.84:0/382907140 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe868142a60 con 0x7fe868146130 2024-03-29T10:34:53.655 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 10 client.?.objecter ms_handle_connect 0x7fe8681da7a0 2024-03-29T10:34:53.655 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:53.655 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 1 -- 172.21.15.84:0/382907140 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1846140430 0 0) 0x7fe84c004eb0 con 0x7fe86809ac40 2024-03-29T10:34:53.656 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 1 -- 172.21.15.84:0/382907140 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8681418b0 con 0x7fe86809ac40 2024-03-29T10:34:53.656 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 1 -- 172.21.15.84:0/382907140 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fe858001be0 con 0x7fe86809ac40 2024-03-29T10:34:53.656 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:53.656 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:34:53.656 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 10 monclient(hunting): dump: 2024-03-29T10:34:53.656 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 1 2024-03-29T10:34:53.656 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:53.656 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.656 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.656 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:53.656 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 1 2024-03-29T10:34:53.656 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:53.656 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:53.656 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:53.656 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T10:34:53.657 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:34:53.657 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 1 -- 172.21.15.84:0/382907140 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1945486311 0 0) 0x7fe868141390 con 0x7fe8681da7a0 2024-03-29T10:34:53.657 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:34:53.657 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:53.657 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:53.657 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 10 monclient(hunting): my global_id is 4183 2024-03-29T10:34:53.657 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 1 -- 172.21.15.84:0/382907140 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe84c004eb0 con 0x7fe8681da7a0 2024-03-29T10:34:53.657 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 1 -- 172.21.15.84:0/382907140 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 47382702 0 0) 0x7fe868142a60 con 0x7fe868146130 2024-03-29T10:34:53.657 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 1 -- 172.21.15.84:0/382907140 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe868141390 con 0x7fe868146130 2024-03-29T10:34:53.657 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 1 -- 172.21.15.84:0/382907140 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fe860001a70 con 0x7fe868146130 2024-03-29T10:34:53.657 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:53.657 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:34:53.657 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.655+0000 7fe85e7fc640 10 monclient(hunting): dump: 2024-03-29T10:34:53.657 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 1 2024-03-29T10:34:53.657 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:53.658 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.658 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.658 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:53.658 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 1 2024-03-29T10:34:53.658 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:53.658 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:53.658 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:53.658 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T10:34:53.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe85e7fc640 1 -- 172.21.15.84:0/382907140 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3521641899 0 0) 0x7fe8681418b0 con 0x7fe86809ac40 2024-03-29T10:34:53.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe85e7fc640 1 -- 172.21.15.84:0/382907140 >> v1:172.21.15.84:6790/0 conn(0x7fe868146130 legacy=0x7fe8681e2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:53.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe85e7fc640 1 -- 172.21.15.84:0/382907140 >> v1:172.21.15.86:6789/0 conn(0x7fe8681da7a0 legacy=0x7fe8681451d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:53.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe85e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:34:53.658 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe85e7fc640 1 monclient: found mon.a 2024-03-29T10:34:53.659 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe85e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:53.659 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe85e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:34:53.659 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe85e7fc640 1 -- 172.21.15.84:0/382907140 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe868141a40 con 0x7fe86809ac40 2024-03-29T10:34:53.659 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe85e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:34:53.659 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe85e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:34:53.659 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe85e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:53.659 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe86fbd6640 5 monclient: authenticate success, global_id 4191 2024-03-29T10:34:53.659 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe86fbd6640 10 monclient: _renew_subs 2024-03-29T10:34:53.659 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe86fbd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:34:53.659 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe86fbd6640 1 -- 172.21.15.84:0/382907140 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe868140390 con 0x7fe86809ac40 2024-03-29T10:34:53.659 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe86fbd6640 10 client.4191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:34:53.659 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe86fbd6640 10 monclient: _renew_subs 2024-03-29T10:34:53.659 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe86fbd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:34:53.659 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe86fbd6640 1 -- 172.21.15.84:0/382907140 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe868145ac0 con 0x7fe86809ac40 2024-03-29T10:34:53.660 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe86fbd6640 1 librados: init done 2024-03-29T10:34:53.660 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe85e7fc640 1 -- 172.21.15.84:0/382907140 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe858004b00 con 0x7fe86809ac40 2024-03-29T10:34:53.660 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe85e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:53.660 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe85e7fc640 1 -- 172.21.15.84:0/382907140 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fe858004e90 con 0x7fe86809ac40 2024-03-29T10:34:53.660 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe85e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:53.660 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe85e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:34:53.660 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe85e7fc640 10 monclient: dump: 2024-03-29T10:34:53.660 INFO:teuthology.orchestra.run.smithi084.stderr:epoch 1 2024-03-29T10:34:53.660 INFO:teuthology.orchestra.run.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:53.660 INFO:teuthology.orchestra.run.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.660 INFO:teuthology.orchestra.run.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:53.660 INFO:teuthology.orchestra.run.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:53.660 INFO:teuthology.orchestra.run.smithi084.stderr:election_strategy: 1 2024-03-29T10:34:53.660 INFO:teuthology.orchestra.run.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:53.661 INFO:teuthology.orchestra.run.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:53.661 INFO:teuthology.orchestra.run.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:53.661 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T10:34:53.661 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe85ffff640 4 set_mon_vals no callback set 2024-03-29T10:34:53.661 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe85e7fc640 1 -- 172.21.15.84:0/382907140 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe858005440 con 0x7fe86809ac40 2024-03-29T10:34:53.661 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe85e7fc640 1 -- 172.21.15.84:0/382907140 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7fe858006620 con 0x7fe86809ac40 2024-03-29T10:34:53.661 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe85e7fc640 10 client.4191.objecter ms_dispatch 0x7fe868000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:34:53.661 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe85e7fc640 3 client.4191.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:34:53.661 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe85e7fc640 3 client.4191.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:34:53.661 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.656+0000 7fe85e7fc640 20 client.4191.objecter dump_active .. 0 homeless 2024-03-29T10:34:53.661 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.657+0000 7fe86fbd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:53.661 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.657+0000 7fe86fbd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:53.661 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.657+0000 7fe86fbd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:34:53.662 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.657+0000 7fe86fbd6640 1 -- 172.21.15.84:0/382907140 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe850005080 con 0x7fe86809ac40 2024-03-29T10:34:53.662 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.659+0000 7fe85e7fc640 1 -- 172.21.15.84:0/382907140 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe8580068c0 con 0x7fe86809ac40 2024-03-29T10:34:53.662 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.659+0000 7fe85e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:53.662 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.659+0000 7fe85e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:34:53.662 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-29T10:34:53.668+0000 7f4e58c21040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:34:53.683 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T10:34:53.697+0000 7f187bc42040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:34:53.698 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-29T10:34:53.713+0000 7f4e58c21040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:34:53.706 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.719+0000 7fe86fbd6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:34:53.706 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.719+0000 7fe86fbd6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:34:53.706 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.719+0000 7fe86fbd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:34:53.706 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.719+0000 7fe86fbd6640 1 -- 172.21.15.84:0/382907140 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe850005330 con 0x7fe86809ac40 2024-03-29T10:34:53.706 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.720+0000 7fe85e7fc640 1 -- 172.21.15.84:0/382907140 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7fe858006ae0 con 0x7fe86809ac40 2024-03-29T10:34:53.706 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.720+0000 7fe85e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:34:53.707 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.720+0000 7fe85e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:34:53.707 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-03-29T10:34:53.707 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:34:53.708 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.721+0000 7fe86fbd6640 10 librados: watch_flush enter 2024-03-29T10:34:53.708 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.721+0000 7fe86fbd6640 10 librados: watch_flush exit 2024-03-29T10:34:53.708 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.721+0000 7fe86fbd6640 20 client.4191.objecter shutdown clearing up homeless session... 2024-03-29T10:34:53.708 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.721+0000 7fe86fbd6640 10 client.4191.objecter successfully canceled tick 2024-03-29T10:34:53.708 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.721+0000 7fe86fbd6640 10 monclient: shutdown 2024-03-29T10:34:53.708 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.721+0000 7fe86fbd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:53.708 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.721+0000 7fe86fbd6640 1 -- 172.21.15.84:0/382907140 >> v1:172.21.15.84:6789/0 conn(0x7fe86809ac40 legacy=0x7fe8681da090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:53.709 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.721+0000 7fe86fbd6640 1 -- 172.21.15.84:0/382907140 shutdown_connections 2024-03-29T10:34:53.709 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.721+0000 7fe86fbd6640 1 -- 172.21.15.84:0/382907140 >> 172.21.15.84:0/382907140 conn(0x7fe868066b00 msgr2=0x7fe8680b7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:53.709 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.721+0000 7fe86fbd6640 1 -- 172.21.15.84:0/382907140 shutdown_connections 2024-03-29T10:34:53.709 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.721+0000 7fe86fbd6640 1 -- 172.21.15.84:0/382907140 wait complete. 2024-03-29T10:34:53.709 INFO:teuthology.orchestra.run.smithi084.stderr:2024-03-29T10:34:53.721+0000 7fe86fbd6640 1 librados: shutdown 2024-03-29T10:34:53.717 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T10:34:53.718 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T10:34:53.799 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T10:34:53.812+0000 7f187bc42040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:34:53.809 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-29T10:34:53.824+0000 7f4e58c21040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:34:53.895 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T10:34:53.908+0000 7f187bc42040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:34:53.905 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-29T10:34:53.919+0000 7f4e58c21040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:34:54.048 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T10:34:54.062+0000 7f187bc42040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:34:54.057 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-29T10:34:54.072+0000 7f4e58c21040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:34:54.090 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T10:34:54.104+0000 7f187bc42040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:34:54.099 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-29T10:34:54.114+0000 7f4e58c21040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:34:54.112 INFO:tasks.ceph.osd.2.smithi084.stderr:2024-03-29T10:34:54.126+0000 7f7db5e1e640 -1 Falling back to public interface 2024-03-29T10:34:54.116 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-03-29T10:34:54.131+0000 7fd3a0c11640 -1 Falling back to public interface 2024-03-29T10:34:54.117 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-03-29T10:34:54.131+0000 7f6789dfa640 -1 Falling back to public interface 2024-03-29T10:34:54.117 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-03-29T10:34:54.131+0000 7fd39f528640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6800/0: (98) Address already in use 2024-03-29T10:34:54.118 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-03-29T10:34:54.131+0000 7fd39f528640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6801/0: (98) Address already in use 2024-03-29T10:34:54.118 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-03-29T10:34:54.132+0000 7fd39f528640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6804/0: (98) Address already in use 2024-03-29T10:34:54.133 INFO:tasks.ceph.osd.0.smithi084.stderr:2024-03-29T10:34:54.147+0000 7fed07214640 -1 Falling back to public interface 2024-03-29T10:34:54.134 INFO:tasks.ceph.osd.1.smithi084.stderr:2024-03-29T10:34:54.147+0000 7f38ea80d640 -1 Falling back to public interface 2024-03-29T10:34:54.134 INFO:tasks.ceph.osd.1.smithi084.stderr:2024-03-29T10:34:54.147+0000 7f38e9128640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6804/0: (98) Address already in use 2024-03-29T10:34:54.135 INFO:tasks.ceph.osd.6.smithi086.stderr:2024-03-29T10:34:54.150+0000 7fb0b9149640 -1 Falling back to public interface 2024-03-29T10:34:54.136 INFO:tasks.ceph.osd.7.smithi086.stderr:2024-03-29T10:34:54.150+0000 7f6171b71640 -1 Falling back to public interface 2024-03-29T10:34:54.136 INFO:tasks.ceph.osd.7.smithi086.stderr:2024-03-29T10:34:54.150+0000 7f6170b20640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6808/0: (98) Address already in use 2024-03-29T10:34:54.141 INFO:tasks.ceph.osd.3.smithi084.stderr:2024-03-29T10:34:54.155+0000 7fa95c4f9640 -1 Falling back to public interface 2024-03-29T10:34:54.246 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T10:34:54.259+0000 7f187bc42040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:34:54.255 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-29T10:34:54.270+0000 7f4e58c21040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:34:54.523 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:34:54.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.628+0000 7f0cf7e25640 10 monclient: get_monmap_and_config 2024-03-29T10:34:54.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.628+0000 7f0cf7e25640 10 monclient: build_initial_monmap 2024-03-29T10:34:54.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.628+0000 7f0cf7e25640 10 monclient: monmap: 2024-03-29T10:34:54.614 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:34:54.614 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:54.614 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:54.629373+0000 2024-03-29T10:34:54.614 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:54.629373+0000 2024-03-29T10:34:54.614 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:54.614 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:34:54.614 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:34:54.614 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:34:54.614 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:34:54.614 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:34:54.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.629+0000 7f0cf7e25640 1 Processor -- start 2024-03-29T10:34:54.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.629+0000 7f0cf7e25640 1 -- start start 2024-03-29T10:34:54.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.629+0000 7f0cf7e25640 10 monclient: init 2024-03-29T10:34:54.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.629+0000 7f0cf7e25640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:54.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.629+0000 7f0cf7e25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:34:54.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.629+0000 7f0cf7e25640 10 monclient(hunting): picked mon.noname-a con 0x7f0cf01457f0 addr v1:172.21.15.84:6789/0 2024-03-29T10:34:54.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.629+0000 7f0cf7e25640 10 monclient(hunting): picked mon.noname-c con 0x7f0cf013f650 addr v1:172.21.15.86:6789/0 2024-03-29T10:34:54.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.629+0000 7f0cf7e25640 10 monclient(hunting): picked mon.noname-b con 0x7f0cf00a9d70 addr v1:172.21.15.84:6790/0 2024-03-29T10:34:54.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.629+0000 7f0cf7e25640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf00aaef0 con 0x7f0cf01457f0 2024-03-29T10:34:54.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.629+0000 7f0cf7e25640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf0145bd0 con 0x7f0cf013f650 2024-03-29T10:34:54.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.629+0000 7f0cf7e25640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf0145eb0 con 0x7f0cf00a9d70 2024-03-29T10:34:54.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.629+0000 7f0cf7e25640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:54.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.629+0000 7f0cf7e25640 10 monclient(hunting): authenticate will time out at 796.893000s 2024-03-29T10:34:54.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.629+0000 7f0cf5bc1640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f0cf01457f0 0x7f0cf013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:56236/0 (socket says 172.21.15.84:56236) 2024-03-29T10:34:54.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.629+0000 7f0cf5bc1640 1 -- 172.21.15.84:0/2140992852 learned_addr learned my addr 172.21.15.84:0/2140992852 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:34:54.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 1 -- 172.21.15.84:0/2140992852 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2865397186 0 0) 0x7f0cf00aaef0 con 0x7f0cf01457f0 2024-03-29T10:34:54.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 10 monclient(hunting): _negotiate 2024-03-29T10:34:54.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:54.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:54.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 10 monclient(hunting): my global_id is 4194 2024-03-29T10:34:54.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 1 -- 172.21.15.84:0/2140992852 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cdc003520 con 0x7f0cf01457f0 2024-03-29T10:34:54.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 1 -- 172.21.15.84:0/2140992852 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1225735534 0 0) 0x7f0cf0145eb0 con 0x7f0cf00a9d70 2024-03-29T10:34:54.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 10 monclient(hunting): _negotiate 2024-03-29T10:34:54.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:54.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:54.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 10 monclient(hunting): my global_id is 4190 2024-03-29T10:34:54.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 1 -- 172.21.15.84:0/2140992852 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cf00aaef0 con 0x7f0cf00a9d70 2024-03-29T10:34:54.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 1 -- 172.21.15.84:0/2140992852 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3468917349 0 0) 0x7f0cf0145bd0 con 0x7f0cf013f650 2024-03-29T10:34:54.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 10 monclient(hunting): _negotiate 2024-03-29T10:34:54.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:54.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:54.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 10 monclient(hunting): my global_id is 4186 2024-03-29T10:34:54.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 1 -- 172.21.15.84:0/2140992852 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cf0145eb0 con 0x7f0cf013f650 2024-03-29T10:34:54.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 1 -- 172.21.15.84:0/2140992852 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1379525965 0 0) 0x7f0cf00aaef0 con 0x7f0cf00a9d70 2024-03-29T10:34:54.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 1 -- 172.21.15.84:0/2140992852 >> v1:172.21.15.86:6789/0 conn(0x7f0cf013f650 legacy=0x7f0cf00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:54.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 1 -- 172.21.15.84:0/2140992852 >> v1:172.21.15.84:6789/0 conn(0x7f0cf01457f0 legacy=0x7f0cf013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:54.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 10 monclient: _finish_hunting 0 2024-03-29T10:34:54.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 1 monclient: found mon.noname-b 2024-03-29T10:34:54.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:54.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:34:54.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 1 -- 172.21.15.84:0/2140992852 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf0146340 con 0x7f0cf00a9d70 2024-03-29T10:34:54.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 10 monclient: _finish_auth 0 2024-03-29T10:34:54.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 10 monclient: _check_auth_tickets 2024-03-29T10:34:54.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:54.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf7e25640 5 monclient: authenticate success, global_id 4190 2024-03-29T10:34:54.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 1 -- 172.21.15.84:0/2140992852 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0ce8001920 con 0x7f0cf00a9d70 2024-03-29T10:34:54.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf7e25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:54.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:54.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:34:54.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 10 monclient: dump: 2024-03-29T10:34:54.627 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:34:54.627 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:54.627 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:54.627 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:54.627 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:54.628 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:34:54.628 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:54.628 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:54.628 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:54.628 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:34:54.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf7e25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:34:54.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 1 -- 172.21.15.84:0/2140992852 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ce8001a80 con 0x7f0cf00a9d70 2024-03-29T10:34:54.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:54.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf7e25640 10 monclient: get_monmap_and_config success 2024-03-29T10:34:54.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 1 -- 172.21.15.84:0/2140992852 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0ce8001ed0 con 0x7f0cf00a9d70 2024-03-29T10:34:54.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:54.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf7e25640 4 set_mon_vals no callback set 2024-03-29T10:34:54.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:34:54.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.630+0000 7f0cf4bbf640 10 monclient: dump: 2024-03-29T10:34:54.629 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:34:54.629 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:54.629 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:54.629 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:54.629 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:54.629 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:34:54.629 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:54.630 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:54.630 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:54.630 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:34:54.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.631+0000 7f0cf7e25640 10 monclient: shutdown 2024-03-29T10:34:54.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.631+0000 7f0cf7e25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:54.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.631+0000 7f0cf7e25640 1 -- 172.21.15.84:0/2140992852 >> v1:172.21.15.84:6790/0 conn(0x7f0cf00a9d70 legacy=0x7f0cf014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:54.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.631+0000 7f0cf7e25640 1 -- 172.21.15.84:0/2140992852 shutdown_connections 2024-03-29T10:34:54.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.631+0000 7f0cf7e25640 1 -- 172.21.15.84:0/2140992852 >> 172.21.15.84:0/2140992852 conn(0x7f0cf0066a00 msgr2=0x7f0cf0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:54.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.631+0000 7f0cf7e25640 1 -- 172.21.15.84:0/2140992852 shutdown_connections 2024-03-29T10:34:54.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.631+0000 7f0cf7e25640 1 -- 172.21.15.84:0/2140992852 wait complete. 2024-03-29T10:34:54.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.632+0000 7f0cf7e25640 10 monclient: build_initial_monmap 2024-03-29T10:34:54.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.632+0000 7f0cf7e25640 10 monclient: monmap: 2024-03-29T10:34:54.631 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:34:54.631 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:34:54.631 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:34:54.631 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:34:54.631 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:34:54.631 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:34:54.632 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:34:54.632 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:34:54.632 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:34:54.632 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:34:54.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.632+0000 7f0cf7e25640 1 librados: starting msgr at 2024-03-29T10:34:54.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.632+0000 7f0cf7e25640 1 librados: starting objecter 2024-03-29T10:34:54.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.632+0000 7f0cf7e25640 1 Processor -- start 2024-03-29T10:34:54.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.632+0000 7f0cf7e25640 1 -- start start 2024-03-29T10:34:54.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.632+0000 7f0cf7e25640 1 librados: setting wanted keys 2024-03-29T10:34:54.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.632+0000 7f0cf7e25640 1 librados: calling monclient init 2024-03-29T10:34:54.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.632+0000 7f0cf7e25640 10 monclient: init 2024-03-29T10:34:54.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.632+0000 7f0cf7e25640 10 monclient: _reopen_session rank -1 2024-03-29T10:34:54.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.632+0000 7f0cf7e25640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:34:54.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.632+0000 7f0cf7e25640 10 monclient(hunting): picked mon.noname-c con 0x7f0cf00a9d70 addr v1:172.21.15.84:6790/0 2024-03-29T10:34:54.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.632+0000 7f0cf7e25640 10 monclient(hunting): picked mon.noname-a con 0x7f0cf013f650 addr v1:172.21.15.84:6789/0 2024-03-29T10:34:54.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.632+0000 7f0cf7e25640 10 monclient(hunting): picked mon.noname-b con 0x7f0cf01457f0 addr v1:172.21.15.86:6789/0 2024-03-29T10:34:54.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.632+0000 7f0cf7e25640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf01d7790 con 0x7f0cf013f650 2024-03-29T10:34:54.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0cf7e25640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf01d8c50 con 0x7f0cf01457f0 2024-03-29T10:34:54.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0cf7e25640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf01d7c90 con 0x7f0cf00a9d70 2024-03-29T10:34:54.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0cf7e25640 10 monclient(hunting): _renew_subs 2024-03-29T10:34:54.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0cf7e25640 10 monclient(hunting): authenticate will time out at 796.896130s 2024-03-29T10:34:54.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0cf53c0640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f0cf013f650 0x7f0cf01399a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:56246/0 (socket says 172.21.15.84:56246) 2024-03-29T10:34:54.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0cf53c0640 1 -- 172.21.15.84:0/2569828982 learned_addr learned my addr 172.21.15.84:0/2569828982 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:34:54.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 client.?.objecter ms_handle_connect 0x7f0cf00a9d70 2024-03-29T10:34:54.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:54.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 client.?.objecter ms_handle_connect 0x7f0cf013f650 2024-03-29T10:34:54.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:54.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 client.?.objecter ms_handle_connect 0x7f0cf01457f0 2024-03-29T10:34:54.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:34:54.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 1 -- 172.21.15.84:0/2569828982 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2538722530 0 0) 0x7f0cf01d7c90 con 0x7f0cf00a9d70 2024-03-29T10:34:54.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:34:54.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:54.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:54.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 monclient(hunting): my global_id is 4193 2024-03-29T10:34:54.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 1 -- 172.21.15.84:0/2569828982 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cd4004eb0 con 0x7f0cf00a9d70 2024-03-29T10:34:54.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 1 -- 172.21.15.84:0/2569828982 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4047502378 0 0) 0x7f0cf01d7790 con 0x7f0cf013f650 2024-03-29T10:34:54.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:34:54.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:54.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:54.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 monclient(hunting): my global_id is 4197 2024-03-29T10:34:54.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 1 -- 172.21.15.84:0/2569828982 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cf01d7c90 con 0x7f0cf013f650 2024-03-29T10:34:54.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 1 -- 172.21.15.84:0/2569828982 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2607018513 0 0) 0x7f0cf01d8c50 con 0x7f0cf01457f0 2024-03-29T10:34:54.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:34:54.636 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:34:54.636 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:34:54.636 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 monclient(hunting): my global_id is 4189 2024-03-29T10:34:54.636 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 1 -- 172.21.15.84:0/2569828982 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cf01d7790 con 0x7f0cf01457f0 2024-03-29T10:34:54.636 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 1 -- 172.21.15.84:0/2569828982 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4265913113 0 0) 0x7f0cd4004eb0 con 0x7f0cf00a9d70 2024-03-29T10:34:54.636 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 1 -- 172.21.15.84:0/2569828982 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cf01d8c50 con 0x7f0cf00a9d70 2024-03-29T10:34:54.636 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 1 -- 172.21.15.84:0/2569828982 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0cec0017f0 con 0x7f0cf00a9d70 2024-03-29T10:34:54.636 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:54.636 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:34:54.636 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 monclient(hunting): dump: 2024-03-29T10:34:54.636 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:34:54.636 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:54.636 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:54.637 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:54.637 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:54.637 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:34:54.637 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:54.637 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:54.637 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:54.637 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:34:54.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:34:54.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 1 -- 172.21.15.84:0/2569828982 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2201095697 0 0) 0x7f0cf01d7c90 con 0x7f0cf013f650 2024-03-29T10:34:54.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 1 -- 172.21.15.84:0/2569828982 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cd4004eb0 con 0x7f0cf013f650 2024-03-29T10:34:54.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 1 -- 172.21.15.84:0/2569828982 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 782561229 0 0) 0x7f0cf01d7790 con 0x7f0cf01457f0 2024-03-29T10:34:54.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 1 -- 172.21.15.84:0/2569828982 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cf01d7c90 con 0x7f0cf01457f0 2024-03-29T10:34:54.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 1 -- 172.21.15.84:0/2569828982 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0ce0001970 con 0x7f0cf013f650 2024-03-29T10:34:54.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:54.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:34:54.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 monclient(hunting): dump: 2024-03-29T10:34:54.638 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:34:54.638 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:54.638 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:54.638 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:54.638 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:54.638 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:34:54.638 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:54.638 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:54.638 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:54.638 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:34:54.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 1 -- 172.21.15.84:0/2569828982 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0ce8001d90 con 0x7f0cf01457f0 2024-03-29T10:34:54.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:54.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:34:54.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 monclient(hunting): dump: 2024-03-29T10:34:54.639 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:34:54.639 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:54.639 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:54.639 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:54.639 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:54.639 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:34:54.639 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:54.639 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:54.639 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:54.639 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:34:54.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 1 -- 172.21.15.84:0/2569828982 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 867544223 0 0) 0x7f0cf01d8c50 con 0x7f0cf00a9d70 2024-03-29T10:34:54.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 1 -- 172.21.15.84:0/2569828982 >> v1:172.21.15.86:6789/0 conn(0x7f0cf01457f0 legacy=0x7f0cf013a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:54.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 1 -- 172.21.15.84:0/2569828982 >> v1:172.21.15.84:6789/0 conn(0x7f0cf013f650 legacy=0x7f0cf01399a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:54.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:34:54.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 1 monclient: found mon.c 2024-03-29T10:34:54.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:34:54.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:34:54.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 1 -- 172.21.15.84:0/2569828982 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf01d7e20 con 0x7f0cf00a9d70 2024-03-29T10:34:54.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 monclient: _finish_auth 0 2024-03-29T10:34:54.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:34:54.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:34:54.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0cf7e25640 5 monclient: authenticate success, global_id 4193 2024-03-29T10:34:54.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0cf7e25640 10 monclient: _renew_subs 2024-03-29T10:34:54.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0cf7e25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:34:54.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0cf7e25640 1 -- 172.21.15.84:0/2569828982 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf0142350 con 0x7f0cf00a9d70 2024-03-29T10:34:54.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0cf7e25640 10 client.4193.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:34:54.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0cf7e25640 10 monclient: _renew_subs 2024-03-29T10:34:54.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0cf7e25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:34:54.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0cf7e25640 1 -- 172.21.15.84:0/2569828982 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf00ae240 con 0x7f0cf00a9d70 2024-03-29T10:34:54.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.633+0000 7f0ce67fc640 1 -- 172.21.15.84:0/2569828982 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cec0047e0 con 0x7f0cf00a9d70 2024-03-29T10:34:54.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.634+0000 7f0cf7e25640 1 librados: init done 2024-03-29T10:34:54.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.634+0000 7f0ce67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:34:54.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.634+0000 7f0ce7fff640 4 set_mon_vals no callback set 2024-03-29T10:34:54.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.634+0000 7f0ce67fc640 1 -- 172.21.15.84:0/2569828982 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0cec001d80 con 0x7f0cf00a9d70 2024-03-29T10:34:54.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.634+0000 7f0ce67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:34:54.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.634+0000 7f0ce67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:34:54.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.634+0000 7f0ce67fc640 10 monclient: dump: 2024-03-29T10:34:54.642 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:34:54.642 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:34:54.642 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:54.642 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:34:54.642 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:34:54.642 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:34:54.642 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:34:54.642 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:34:54.642 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:34:54.642 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:34:54.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.634+0000 7f0ce67fc640 1 -- 172.21.15.84:0/2569828982 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0cec004f00 con 0x7f0cf00a9d70 2024-03-29T10:34:54.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.634+0000 7f0ce67fc640 1 -- 172.21.15.84:0/2569828982 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f0cec006800 con 0x7f0cf00a9d70 2024-03-29T10:34:54.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.634+0000 7f0ce67fc640 10 client.4193.objecter ms_dispatch 0x7f0cf0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:34:54.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.634+0000 7f0ce67fc640 3 client.4193.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:34:54.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.634+0000 7f0ce67fc640 3 client.4193.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:34:54.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.635+0000 7f0ce67fc640 20 client.4193.objecter dump_active .. 0 homeless 2024-03-29T10:34:54.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.635+0000 7f0cf7e25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:54.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.635+0000 7f0cf7e25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:54.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.635+0000 7f0cf7e25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:34:54.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.635+0000 7f0cf7e25640 1 -- 172.21.15.84:0/2569828982 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf01426f0 con 0x7f0cf00a9d70 2024-03-29T10:34:54.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.637+0000 7f0ce67fc640 1 -- 172.21.15.84:0/2569828982 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0cec006a10 con 0x7f0cf00a9d70 2024-03-29T10:34:54.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.637+0000 7f0ce67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:34:54.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.637+0000 7f0ce67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:34:54.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.698+0000 7f0cf7e25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:34:54.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.698+0000 7f0cf7e25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:34:54.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.698+0000 7f0cf7e25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:34:54.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.698+0000 7f0cf7e25640 1 -- 172.21.15.84:0/2569828982 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0cf0075f10 con 0x7f0cf00a9d70 2024-03-29T10:34:54.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.698+0000 7f0ce67fc640 1 -- 172.21.15.84:0/2569828982 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f0cec006bb0 con 0x7f0cf00a9d70 2024-03-29T10:34:54.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.698+0000 7f0ce67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:34:54.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.698+0000 7f0ce67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:34:54.685 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:34:54.685 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:34:54.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.699+0000 7f0cf7e25640 10 librados: watch_flush enter 2024-03-29T10:34:54.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.699+0000 7f0cf7e25640 10 librados: watch_flush exit 2024-03-29T10:34:54.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.699+0000 7f0cf7e25640 20 client.4193.objecter shutdown clearing up homeless session... 2024-03-29T10:34:54.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.699+0000 7f0cf7e25640 10 client.4193.objecter successfully canceled tick 2024-03-29T10:34:54.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.699+0000 7f0cf7e25640 10 monclient: shutdown 2024-03-29T10:34:54.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.699+0000 7f0cf7e25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:34:54.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.699+0000 7f0cf7e25640 1 -- 172.21.15.84:0/2569828982 >> v1:172.21.15.84:6790/0 conn(0x7f0cf00a9d70 legacy=0x7f0cf01d8540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:34:54.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.699+0000 7f0cf7e25640 1 -- 172.21.15.84:0/2569828982 shutdown_connections 2024-03-29T10:34:54.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.699+0000 7f0cf7e25640 1 -- 172.21.15.84:0/2569828982 >> 172.21.15.84:0/2569828982 conn(0x7f0cf0066a00 msgr2=0x7f0cf0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:34:54.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.699+0000 7f0cf7e25640 1 -- 172.21.15.84:0/2569828982 shutdown_connections 2024-03-29T10:34:54.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.699+0000 7f0cf7e25640 1 -- 172.21.15.84:0/2569828982 wait complete. 2024-03-29T10:34:54.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:34:54.700+0000 7f0cf7e25640 1 librados: shutdown 2024-03-29T10:34:54.695 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:34:55.046 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-29T10:34:55.061+0000 7f4e58c21040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:34:55.058 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T10:34:55.073+0000 7f187bc42040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:34:55.316 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-29T10:34:55.331+0000 7f4e58c21040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:34:55.330 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T10:34:55.345+0000 7f187bc42040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:34:55.368 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-29T10:34:55.383+0000 7f4e58c21040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:34:55.381 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T10:34:55.396+0000 7f187bc42040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:34:55.482 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-29T10:34:55.497+0000 7f4e58c21040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:34:55.496 INFO:tasks.ceph.mgr.y.smithi084.stderr:2024-03-29T10:34:55.511+0000 7f187bc42040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:34:56.497 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-29T10:34:56.511+0000 7f4df95dc640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T10:34:56.497 INFO:tasks.ceph.mgr.x.smithi086.stderr:2024-03-29T10:34:56.512+0000 7f4df95dc640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T10:34:57.566 INFO:tasks.ceph.osd.4.smithi086.stderr:2024-03-29T10:34:57.581+0000 7f6789dfa640 -1 osd.4 0 log_to_monitors true 2024-03-29T10:34:57.584 INFO:tasks.ceph.osd.2.smithi084.stderr:2024-03-29T10:34:57.596+0000 7f7db5e1e640 -1 osd.2 0 log_to_monitors true 2024-03-29T10:34:57.605 INFO:tasks.ceph.osd.7.smithi086.stderr:2024-03-29T10:34:57.619+0000 7f6171b71640 -1 osd.7 0 log_to_monitors true 2024-03-29T10:34:57.646 INFO:tasks.ceph.osd.0.smithi084.stderr:2024-03-29T10:34:57.661+0000 7fed07214640 -1 osd.0 0 log_to_monitors true 2024-03-29T10:34:57.651 INFO:tasks.ceph.osd.6.smithi086.stderr:2024-03-29T10:34:57.665+0000 7fb0b9149640 -1 osd.6 0 log_to_monitors true 2024-03-29T10:34:57.657 INFO:tasks.ceph.osd.5.smithi086.stderr:2024-03-29T10:34:57.670+0000 7fd3a0c11640 -1 osd.5 0 log_to_monitors true 2024-03-29T10:34:57.662 INFO:tasks.ceph.osd.1.smithi084.stderr:2024-03-29T10:34:57.676+0000 7f38ea80d640 -1 osd.1 0 log_to_monitors true 2024-03-29T10:34:57.664 INFO:tasks.ceph.osd.3.smithi084.stderr:2024-03-29T10:34:57.680+0000 7fa95c4f9640 -1 osd.3 0 log_to_monitors true 2024-03-29T10:35:01.502 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:35:01.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.671+0000 7f013af2a640 10 monclient: get_monmap_and_config 2024-03-29T10:35:01.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.671+0000 7f013af2a640 10 monclient: build_initial_monmap 2024-03-29T10:35:01.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.671+0000 7f013af2a640 10 monclient: monmap: 2024-03-29T10:35:01.657 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:35:01.657 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:01.657 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:35:01.672846+0000 2024-03-29T10:35:01.657 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:35:01.672846+0000 2024-03-29T10:35:01.657 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:01.657 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:01.658 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:35:01.658 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:35:01.658 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:35:01.658 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:01.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.672+0000 7f013af2a640 1 Processor -- start 2024-03-29T10:35:01.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.673+0000 7f013af2a640 1 -- start start 2024-03-29T10:35:01.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.673+0000 7f013af2a640 10 monclient: init 2024-03-29T10:35:01.662 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.674+0000 7f013af2a640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:01.662 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.674+0000 7f013af2a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:35:01.663 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.674+0000 7f013af2a640 10 monclient(hunting): picked mon.noname-c con 0x7f012c006a70 addr v1:172.21.15.86:6789/0 2024-03-29T10:35:01.663 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.674+0000 7f013af2a640 10 monclient(hunting): picked mon.noname-b con 0x7f012c02b100 addr v1:172.21.15.84:6790/0 2024-03-29T10:35:01.663 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.674+0000 7f013af2a640 10 monclient(hunting): picked mon.noname-a con 0x7f012c029c30 addr v1:172.21.15.84:6789/0 2024-03-29T10:35:01.663 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.674+0000 7f013af2a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f012c02a270 con 0x7f012c029c30 2024-03-29T10:35:01.663 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.674+0000 7f013af2a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f012c02c440 con 0x7f012c006a70 2024-03-29T10:35:01.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.674+0000 7f013af2a640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f012c0c9440 con 0x7f012c02b100 2024-03-29T10:35:01.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.674+0000 7f013af2a640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:01.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.674+0000 7f013af2a640 10 monclient(hunting): authenticate will time out at 803.937276s 2024-03-29T10:35:01.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.674+0000 7f0139727640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f012c02b100 0x7f012c0cf100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:60830/0 (socket says 172.21.15.84:60830) 2024-03-29T10:35:01.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.674+0000 7f0139727640 1 -- 172.21.15.84:0/33692633 learned_addr learned my addr 172.21.15.84:0/33692633 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:35:01.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.675+0000 7f0138f26640 1 -- 172.21.15.84:0/33692633 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310407204 0 0) 0x7f012c0c9440 con 0x7f012c02b100 2024-03-29T10:35:01.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.675+0000 7f0138f26640 10 monclient(hunting): _negotiate 2024-03-29T10:35:01.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.675+0000 7f0138f26640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:01.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.675+0000 7f0138f26640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:01.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.675+0000 7f0138f26640 10 monclient(hunting): my global_id is 4235 2024-03-29T10:35:01.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.675+0000 7f0138f26640 1 -- 172.21.15.84:0/33692633 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0120003520 con 0x7f012c02b100 2024-03-29T10:35:01.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.675+0000 7f0138f26640 1 -- 172.21.15.84:0/33692633 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3560674878 0 0) 0x7f012c02c440 con 0x7f012c006a70 2024-03-29T10:35:01.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.675+0000 7f0138f26640 10 monclient(hunting): _negotiate 2024-03-29T10:35:01.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.675+0000 7f0138f26640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:01.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.675+0000 7f0138f26640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:01.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.675+0000 7f0138f26640 10 monclient(hunting): my global_id is 4234 2024-03-29T10:35:01.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.675+0000 7f0138f26640 1 -- 172.21.15.84:0/33692633 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f012c0c9440 con 0x7f012c006a70 2024-03-29T10:35:01.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.676+0000 7f0138f26640 1 -- 172.21.15.84:0/33692633 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 635291454 0 0) 0x7f0120003520 con 0x7f012c02b100 2024-03-29T10:35:01.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.676+0000 7f0138f26640 1 -- 172.21.15.84:0/33692633 >> v1:172.21.15.86:6789/0 conn(0x7f012c006a70 legacy=0x7f012c0c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:01.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.676+0000 7f0138f26640 1 -- 172.21.15.84:0/33692633 >> v1:172.21.15.84:6789/0 conn(0x7f012c029c30 legacy=0x7f012c0d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:01.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.676+0000 7f0138f26640 10 monclient: _finish_hunting 0 2024-03-29T10:35:01.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.676+0000 7f0138f26640 1 monclient: found mon.noname-b 2024-03-29T10:35:01.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.676+0000 7f0138f26640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:01.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.676+0000 7f0138f26640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:35:01.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.676+0000 7f0138f26640 1 -- 172.21.15.84:0/33692633 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f012c0c95d0 con 0x7f012c02b100 2024-03-29T10:35:01.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.676+0000 7f0138f26640 10 monclient: _finish_auth 0 2024-03-29T10:35:01.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.676+0000 7f0138f26640 10 monclient: _check_auth_tickets 2024-03-29T10:35:01.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.676+0000 7f0138f26640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:01.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.676+0000 7f0138f26640 1 -- 172.21.15.84:0/33692633 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0124001690 con 0x7f012c02b100 2024-03-29T10:35:01.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.676+0000 7f0138f26640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:01.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.676+0000 7f0138f26640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:35:01.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.676+0000 7f0138f26640 10 monclient: dump: 2024-03-29T10:35:01.666 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:01.666 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:01.666 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:01.666 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:01.666 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:01.667 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:01.667 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:01.667 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:01.667 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:01.667 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:01.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.676+0000 7f013af2a640 5 monclient: authenticate success, global_id 4235 2024-03-29T10:35:01.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.676+0000 7f013af2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:35:01.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.678+0000 7f0138f26640 1 -- 172.21.15.84:0/33692633 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01240017f0 con 0x7f012c02b100 2024-03-29T10:35:01.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.678+0000 7f0138f26640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:01.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.678+0000 7f013af2a640 10 monclient: get_monmap_and_config success 2024-03-29T10:35:01.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.678+0000 7f013af2a640 4 set_mon_vals no callback set 2024-03-29T10:35:01.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.678+0000 7f0138f26640 1 -- 172.21.15.84:0/33692633 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0124001c40 con 0x7f012c02b100 2024-03-29T10:35:01.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.678+0000 7f0138f26640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:01.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.678+0000 7f0138f26640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:35:01.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.678+0000 7f0138f26640 10 monclient: dump: 2024-03-29T10:35:01.678 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:01.678 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:01.678 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:01.678 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:01.678 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:01.678 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:01.678 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:01.678 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:01.678 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:01.678 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:01.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.679+0000 7f013af2a640 10 monclient: shutdown 2024-03-29T10:35:01.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.679+0000 7f013af2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:01.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.679+0000 7f013af2a640 1 -- 172.21.15.84:0/33692633 >> v1:172.21.15.84:6790/0 conn(0x7f012c02b100 legacy=0x7f012c0cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:01.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.679+0000 7f013af2a640 1 -- 172.21.15.84:0/33692633 shutdown_connections 2024-03-29T10:35:01.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.679+0000 7f013af2a640 1 -- 172.21.15.84:0/33692633 >> 172.21.15.84:0/33692633 conn(0x7f012c0058f0 msgr2=0x7f012c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:01.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.679+0000 7f013af2a640 1 -- 172.21.15.84:0/33692633 shutdown_connections 2024-03-29T10:35:01.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.679+0000 7f013af2a640 1 -- 172.21.15.84:0/33692633 wait complete. 2024-03-29T10:35:01.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.680+0000 7f013af2a640 10 monclient: build_initial_monmap 2024-03-29T10:35:01.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.680+0000 7f013af2a640 10 monclient: monmap: 2024-03-29T10:35:01.679 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:35:01.679 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:35:01.679 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:35:01.679 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:35:01.679 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:01.679 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:01.679 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:35:01.679 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:35:01.680 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:35:01.680 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:01.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.681+0000 7f013af2a640 1 librados: starting msgr at 2024-03-29T10:35:01.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.681+0000 7f013af2a640 1 librados: starting objecter 2024-03-29T10:35:01.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.681+0000 7f013af2a640 1 Processor -- start 2024-03-29T10:35:01.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.681+0000 7f013af2a640 1 -- start start 2024-03-29T10:35:01.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.681+0000 7f013af2a640 1 librados: setting wanted keys 2024-03-29T10:35:01.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.681+0000 7f013af2a640 1 librados: calling monclient init 2024-03-29T10:35:01.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.681+0000 7f013af2a640 10 monclient: init 2024-03-29T10:35:01.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.682+0000 7f013af2a640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:01.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.682+0000 7f013af2a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:35:01.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.682+0000 7f013af2a640 10 monclient(hunting): picked mon.noname-c con 0x7f012c006a70 addr v1:172.21.15.84:6790/0 2024-03-29T10:35:01.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.682+0000 7f013af2a640 10 monclient(hunting): picked mon.noname-a con 0x7f012c029c30 addr v1:172.21.15.84:6789/0 2024-03-29T10:35:01.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.682+0000 7f013af2a640 10 monclient(hunting): picked mon.noname-b con 0x7f012c164970 addr v1:172.21.15.86:6789/0 2024-03-29T10:35:01.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.682+0000 7f013af2a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f012c157170 con 0x7f012c029c30 2024-03-29T10:35:01.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.682+0000 7f013af2a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f012c1573a0 con 0x7f012c164970 2024-03-29T10:35:01.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.682+0000 7f013af2a640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f012c157680 con 0x7f012c006a70 2024-03-29T10:35:01.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.682+0000 7f013af2a640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:01.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.682+0000 7f013af2a640 10 monclient(hunting): authenticate will time out at 803.945903s 2024-03-29T10:35:01.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.682+0000 7f013a729640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f012c164970 0x7f012c16b7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:50818/0 (socket says 172.21.15.84:50818) 2024-03-29T10:35:01.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.682+0000 7f013a729640 1 -- 172.21.15.84:0/1518502824 learned_addr learned my addr 172.21.15.84:0/1518502824 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:35:01.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.683+0000 7f012a7fc640 10 client.?.objecter ms_handle_connect 0x7f012c164970 2024-03-29T10:35:01.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.683+0000 7f012a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:01.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.683+0000 7f012a7fc640 10 client.?.objecter ms_handle_connect 0x7f012c006a70 2024-03-29T10:35:01.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.683+0000 7f012a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:01.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.683+0000 7f012a7fc640 10 client.?.objecter ms_handle_connect 0x7f012c029c30 2024-03-29T10:35:01.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.683+0000 7f012a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:01.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.683+0000 7f012a7fc640 1 -- 172.21.15.84:0/1518502824 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139347465 0 0) 0x7f012c1573a0 con 0x7f012c164970 2024-03-29T10:35:01.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.683+0000 7f012a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:35:01.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.683+0000 7f012a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:01.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.683+0000 7f012a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:01.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.683+0000 7f012a7fc640 10 monclient(hunting): my global_id is 4237 2024-03-29T10:35:01.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.683+0000 7f012a7fc640 1 -- 172.21.15.84:0/1518502824 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0118004eb0 con 0x7f012c164970 2024-03-29T10:35:01.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.683+0000 7f012a7fc640 1 -- 172.21.15.84:0/1518502824 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1720677716 0 0) 0x7f012c157680 con 0x7f012c006a70 2024-03-29T10:35:01.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.683+0000 7f012a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:35:01.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.683+0000 7f012a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:01.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.683+0000 7f012a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:01.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.683+0000 7f012a7fc640 10 monclient(hunting): my global_id is 4238 2024-03-29T10:35:01.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.683+0000 7f012a7fc640 1 -- 172.21.15.84:0/1518502824 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f012c1573a0 con 0x7f012c006a70 2024-03-29T10:35:01.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.684+0000 7f012a7fc640 1 -- 172.21.15.84:0/1518502824 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659505494 0 0) 0x7f012c157170 con 0x7f012c029c30 2024-03-29T10:35:01.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.684+0000 7f012a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:35:01.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.684+0000 7f012a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:01.683 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.684+0000 7f012a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:01.683 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.684+0000 7f012a7fc640 10 monclient(hunting): my global_id is 4245 2024-03-29T10:35:01.683 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.684+0000 7f012a7fc640 1 -- 172.21.15.84:0/1518502824 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f012c157680 con 0x7f012c029c30 2024-03-29T10:35:01.683 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.684+0000 7f012a7fc640 1 -- 172.21.15.84:0/1518502824 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3525107378 0 0) 0x7f0118004eb0 con 0x7f012c164970 2024-03-29T10:35:01.683 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.684+0000 7f012a7fc640 1 -- 172.21.15.84:0/1518502824 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f012c157170 con 0x7f012c164970 2024-03-29T10:35:01.683 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.684+0000 7f012a7fc640 1 -- 172.21.15.84:0/1518502824 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f013404fdd0 con 0x7f012c164970 2024-03-29T10:35:01.683 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.684+0000 7f012a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:01.683 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.684+0000 7f012a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:35:01.683 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.684+0000 7f012a7fc640 10 monclient(hunting): dump: 2024-03-29T10:35:01.683 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:01.683 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:01.683 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:01.683 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:01.683 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:01.684 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:01.684 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:01.684 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:01.684 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:01.684 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:01.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.684+0000 7f012a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:35:01.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.684+0000 7f012a7fc640 1 -- 172.21.15.84:0/1518502824 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 979867114 0 0) 0x7f012c157170 con 0x7f012c164970 2024-03-29T10:35:01.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.684+0000 7f012a7fc640 1 -- 172.21.15.84:0/1518502824 >> v1:172.21.15.84:6790/0 conn(0x7f012c006a70 legacy=0x7f012c155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:01.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.684+0000 7f012a7fc640 1 -- 172.21.15.84:0/1518502824 >> v1:172.21.15.84:6789/0 conn(0x7f012c029c30 legacy=0x7f012c164260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:01.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.684+0000 7f012a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:35:01.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.684+0000 7f012a7fc640 1 monclient: found mon.b 2024-03-29T10:35:01.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.684+0000 7f012a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:01.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.684+0000 7f012a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:35:01.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.684+0000 7f012a7fc640 1 -- 172.21.15.84:0/1518502824 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f012c155b10 con 0x7f012c164970 2024-03-29T10:35:01.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.684+0000 7f012a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:35:01.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.684+0000 7f012a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:35:01.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.684+0000 7f012a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:01.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.685+0000 7f013af2a640 5 monclient: authenticate success, global_id 4237 2024-03-29T10:35:01.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.685+0000 7f013af2a640 10 monclient: _renew_subs 2024-03-29T10:35:01.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.685+0000 7f013af2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:35:01.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.685+0000 7f013af2a640 1 -- 172.21.15.84:0/1518502824 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f012c1563a0 con 0x7f012c164970 2024-03-29T10:35:01.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.685+0000 7f013af2a640 10 client.4237.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:35:01.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.685+0000 7f013af2a640 10 monclient: _renew_subs 2024-03-29T10:35:01.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.685+0000 7f013af2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:35:01.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.685+0000 7f013af2a640 1 -- 172.21.15.84:0/1518502824 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f012c16d0a0 con 0x7f012c164970 2024-03-29T10:35:01.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.685+0000 7f013af2a640 1 librados: init done 2024-03-29T10:35:01.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.686+0000 7f012a7fc640 1 -- 172.21.15.84:0/1518502824 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f013404f400 con 0x7f012c164970 2024-03-29T10:35:01.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.686+0000 7f011ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:01.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.686+0000 7f011ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:01.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.686+0000 7f011ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:35:01.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.686+0000 7f011ffff640 1 -- 172.21.15.84:0/1518502824 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f012c156590 con 0x7f012c164970 2024-03-29T10:35:01.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.686+0000 7f012a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:01.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.686+0000 7f012b7fe640 4 set_mon_vals no callback set 2024-03-29T10:35:01.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.687+0000 7f012a7fc640 1 -- 172.21.15.84:0/1518502824 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0134075f10 con 0x7f012c164970 2024-03-29T10:35:01.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.688+0000 7f012a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:01.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.688+0000 7f012a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:35:01.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.688+0000 7f012a7fc640 10 monclient: dump: 2024-03-29T10:35:01.686 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:01.686 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:01.686 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:01.686 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:01.686 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:01.686 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:01.686 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:01.687 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:01.687 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:01.687 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:01.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.688+0000 7f012a7fc640 1 -- 172.21.15.84:0/1518502824 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f0134067bf0 con 0x7f012c164970 2024-03-29T10:35:01.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.688+0000 7f0139f28640 10 monclient: get_auth_request con 0x7f011806dd00 auth_method 0 2024-03-29T10:35:01.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.688+0000 7f012a7fc640 1 -- 172.21.15.84:0/1518502824 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f01340ff430 con 0x7f012c164970 2024-03-29T10:35:01.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.688+0000 7f012a7fc640 10 client.4237.objecter ms_dispatch 0x7f0134000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:35:01.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.688+0000 7f012a7fc640 3 client.4237.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:35:01.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.688+0000 7f012a7fc640 3 client.4237.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:35:01.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.688+0000 7f012a7fc640 20 client.4237.objecter dump_active .. 0 homeless 2024-03-29T10:35:01.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.689+0000 7f012a7fc640 10 client.4237.objecter ms_handle_connect 0x7f011806dd00 2024-03-29T10:35:01.688 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.694+0000 7f012a7fc640 1 -- 172.21.15.84:0/1518502824 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f01340cad90 con 0x7f012c164970 2024-03-29T10:35:01.688 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.694+0000 7f012a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:01.688 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:01.694+0000 7f012a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:35:02.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:02.060+0000 7f011ffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:02.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:02.060+0000 7f011ffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:02.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:02.060+0000 7f011ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:35:02.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:02.060+0000 7f011ffff640 1 -- 172.21.15.84:0/1518502824 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f012c004cd0 con 0x7f012c164970 2024-03-29T10:35:02.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:02.061+0000 7f012a7fc640 1 -- 172.21.15.84:0/1518502824 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f01340cccc0 con 0x7f012c164970 2024-03-29T10:35:02.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:02.061+0000 7f012a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:02.047 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:02.061+0000 7f012a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:35:02.047 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:35:02.047 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:35:02.050 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:02.065+0000 7f011ffff640 10 librados: watch_flush enter 2024-03-29T10:35:02.050 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:02.065+0000 7f011ffff640 10 librados: watch_flush exit 2024-03-29T10:35:02.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:02.065+0000 7f011ffff640 20 client.4237.objecter shutdown clearing up homeless session... 2024-03-29T10:35:02.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:02.065+0000 7f011ffff640 10 client.4237.objecter successfully canceled tick 2024-03-29T10:35:02.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:02.065+0000 7f011ffff640 1 -- 172.21.15.84:0/1518502824 >> v1:172.21.15.86:6816/3337225337 conn(0x7f011806dd00 legacy=0x7f01180701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:02.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:02.065+0000 7f011ffff640 10 monclient: shutdown 2024-03-29T10:35:02.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:02.065+0000 7f011ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:02.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:02.065+0000 7f011ffff640 1 -- 172.21.15.84:0/1518502824 >> v1:172.21.15.86:6789/0 conn(0x7f012c164970 legacy=0x7f012c16b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:02.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:02.065+0000 7f011ffff640 1 -- 172.21.15.84:0/1518502824 shutdown_connections 2024-03-29T10:35:02.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:02.065+0000 7f011ffff640 1 -- 172.21.15.84:0/1518502824 >> 172.21.15.84:0/1518502824 conn(0x7f012c0058f0 msgr2=0x7f012c0c3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:02.052 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:02.066+0000 7f011ffff640 1 -- 172.21.15.84:0/1518502824 shutdown_connections 2024-03-29T10:35:02.052 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:02.066+0000 7f011ffff640 1 -- 172.21.15.84:0/1518502824 wait complete. 2024-03-29T10:35:02.052 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:02.066+0000 7f011ffff640 1 librados: shutdown 2024-03-29T10:35:02.069 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:35:08.877 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:35:09.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.039+0000 7fb9e9a14640 10 monclient: get_monmap_and_config 2024-03-29T10:35:09.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.040+0000 7fb9e9a14640 10 monclient: build_initial_monmap 2024-03-29T10:35:09.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.040+0000 7fb9e9a14640 10 monclient: monmap: 2024-03-29T10:35:09.025 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:35:09.025 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:09.025 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:35:09.040985+0000 2024-03-29T10:35:09.026 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:35:09.040985+0000 2024-03-29T10:35:09.026 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:09.026 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:09.026 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:35:09.026 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:35:09.026 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:35:09.026 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:09.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.041+0000 7fb9e9a14640 1 Processor -- start 2024-03-29T10:35:09.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.041+0000 7fb9e9a14640 1 -- start start 2024-03-29T10:35:09.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.041+0000 7fb9e9a14640 10 monclient: init 2024-03-29T10:35:09.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.042+0000 7fb9e9a14640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:09.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.042+0000 7fb9e9a14640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:35:09.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.042+0000 7fb9e9a14640 10 monclient(hunting): picked mon.noname-c con 0x7fb9e41457f0 addr v1:172.21.15.86:6789/0 2024-03-29T10:35:09.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.042+0000 7fb9e9a14640 10 monclient(hunting): picked mon.noname-b con 0x7fb9e413f650 addr v1:172.21.15.84:6790/0 2024-03-29T10:35:09.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.042+0000 7fb9e9a14640 10 monclient(hunting): picked mon.noname-a con 0x7fb9e40a9d70 addr v1:172.21.15.84:6789/0 2024-03-29T10:35:09.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.042+0000 7fb9e9a14640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e40aaef0 con 0x7fb9e40a9d70 2024-03-29T10:35:09.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.042+0000 7fb9e9a14640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e4145bd0 con 0x7fb9e41457f0 2024-03-29T10:35:09.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.042+0000 7fb9e27fc640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fb9e413f650 0x7fb9e40b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:37450/0 (socket says 172.21.15.84:37450) 2024-03-29T10:35:09.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.042+0000 7fb9e27fc640 1 -- 172.21.15.84:0/641539119 learned_addr learned my addr 172.21.15.84:0/641539119 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:35:09.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.043+0000 7fb9e9a14640 1 -- 172.21.15.84:0/641539119 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e4145eb0 con 0x7fb9e413f650 2024-03-29T10:35:09.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.043+0000 7fb9e9a14640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:09.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.043+0000 7fb9e9a14640 10 monclient(hunting): authenticate will time out at 811.306426s 2024-03-29T10:35:09.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.043+0000 7fb9e1ffb640 1 -- 172.21.15.84:0/641539119 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2799630556 0 0) 0x7fb9e4145eb0 con 0x7fb9e413f650 2024-03-29T10:35:09.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.043+0000 7fb9e1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:35:09.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.043+0000 7fb9e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:09.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.043+0000 7fb9e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:09.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.043+0000 7fb9e1ffb640 10 monclient(hunting): my global_id is 4241 2024-03-29T10:35:09.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.043+0000 7fb9e1ffb640 1 -- 172.21.15.84:0/641539119 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9cc003520 con 0x7fb9e413f650 2024-03-29T10:35:09.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.043+0000 7fb9e1ffb640 1 -- 172.21.15.84:0/641539119 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2964003066 0 0) 0x7fb9cc003520 con 0x7fb9e413f650 2024-03-29T10:35:09.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.044+0000 7fb9e1ffb640 1 -- 172.21.15.84:0/641539119 >> v1:172.21.15.86:6789/0 conn(0x7fb9e41457f0 legacy=0x7fb9e413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:35:09.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.044+0000 7fb9e1ffb640 1 -- 172.21.15.84:0/641539119 >> v1:172.21.15.84:6789/0 conn(0x7fb9e40a9d70 legacy=0x7fb9e414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:35:09.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.044+0000 7fb9e1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:35:09.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.044+0000 7fb9e1ffb640 1 monclient: found mon.noname-b 2024-03-29T10:35:09.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.044+0000 7fb9e1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:09.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.044+0000 7fb9e1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:35:09.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.044+0000 7fb9e1ffb640 1 -- 172.21.15.84:0/641539119 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e4146340 con 0x7fb9e413f650 2024-03-29T10:35:09.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.045+0000 7fb9e1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:35:09.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.045+0000 7fb9e1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:35:09.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.045+0000 7fb9e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:09.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.045+0000 7fb9e1ffb640 1 -- 172.21.15.84:0/641539119 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fb9d8001690 con 0x7fb9e413f650 2024-03-29T10:35:09.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.045+0000 7fb9e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:09.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.045+0000 7fb9e1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:35:09.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.045+0000 7fb9e1ffb640 10 monclient: dump: 2024-03-29T10:35:09.033 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:09.033 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:09.033 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:09.033 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:09.033 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:09.033 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:09.033 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:09.033 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:09.033 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:09.033 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:09.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.045+0000 7fb9e9a14640 5 monclient: authenticate success, global_id 4241 2024-03-29T10:35:09.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.045+0000 7fb9e9a14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:35:09.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.046+0000 7fb9e1ffb640 1 -- 172.21.15.84:0/641539119 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9d8001920 con 0x7fb9e413f650 2024-03-29T10:35:09.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.046+0000 7fb9e1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:09.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.046+0000 7fb9e1ffb640 1 -- 172.21.15.84:0/641539119 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fb9d8001cb0 con 0x7fb9e413f650 2024-03-29T10:35:09.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.046+0000 7fb9e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:09.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.046+0000 7fb9e1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:35:09.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.046+0000 7fb9e1ffb640 10 monclient: dump: 2024-03-29T10:35:09.034 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:09.034 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:09.034 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:09.034 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:09.034 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:09.034 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:09.034 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:09.034 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:09.034 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:09.035 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:09.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.046+0000 7fb9e9a14640 10 monclient: get_monmap_and_config success 2024-03-29T10:35:09.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.046+0000 7fb9e9a14640 4 set_mon_vals no callback set 2024-03-29T10:35:09.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.047+0000 7fb9e9a14640 10 monclient: shutdown 2024-03-29T10:35:09.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.047+0000 7fb9e9a14640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:09.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.047+0000 7fb9e9a14640 1 -- 172.21.15.84:0/641539119 >> v1:172.21.15.84:6790/0 conn(0x7fb9e413f650 legacy=0x7fb9e40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:09.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.048+0000 7fb9e9a14640 1 -- 172.21.15.84:0/641539119 shutdown_connections 2024-03-29T10:35:09.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.048+0000 7fb9e9a14640 1 -- 172.21.15.84:0/641539119 >> 172.21.15.84:0/641539119 conn(0x7fb9e4066a00 msgr2=0x7fb9e4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:09.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.048+0000 7fb9e9a14640 1 -- 172.21.15.84:0/641539119 shutdown_connections 2024-03-29T10:35:09.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.048+0000 7fb9e9a14640 1 -- 172.21.15.84:0/641539119 wait complete. 2024-03-29T10:35:09.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.049+0000 7fb9e9a14640 10 monclient: build_initial_monmap 2024-03-29T10:35:09.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.049+0000 7fb9e9a14640 10 monclient: monmap: 2024-03-29T10:35:09.041 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:35:09.041 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:35:09.041 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:35:09.041 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:35:09.041 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:09.041 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:09.041 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:35:09.041 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:35:09.041 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:35:09.041 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:09.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.049+0000 7fb9e9a14640 1 librados: starting msgr at 2024-03-29T10:35:09.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.049+0000 7fb9e9a14640 1 librados: starting objecter 2024-03-29T10:35:09.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.049+0000 7fb9e9a14640 1 Processor -- start 2024-03-29T10:35:09.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.050+0000 7fb9e9a14640 1 -- start start 2024-03-29T10:35:09.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.050+0000 7fb9e9a14640 1 librados: setting wanted keys 2024-03-29T10:35:09.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.050+0000 7fb9e9a14640 1 librados: calling monclient init 2024-03-29T10:35:09.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.050+0000 7fb9e9a14640 10 monclient: init 2024-03-29T10:35:09.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.050+0000 7fb9e9a14640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:09.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.051+0000 7fb9e9a14640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:35:09.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.051+0000 7fb9e9a14640 10 monclient(hunting): picked mon.noname-c con 0x7fb9e40a9d70 addr v1:172.21.15.84:6790/0 2024-03-29T10:35:09.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.051+0000 7fb9e2ffd640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fb9e40a9d70 0x7fb9e4137620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:39670/0 (socket says 172.21.15.84:39670) 2024-03-29T10:35:09.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.051+0000 7fb9e2ffd640 1 -- 172.21.15.84:0/2055372592 learned_addr learned my addr 172.21.15.84:0/2055372592 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:35:09.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.051+0000 7fb9cbfff640 10 client.?.objecter ms_handle_connect 0x7fb9e40a9d70 2024-03-29T10:35:09.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.051+0000 7fb9cbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:09.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.051+0000 7fb9e9a14640 10 monclient(hunting): picked mon.noname-b con 0x7fb9e413f650 addr v1:172.21.15.86:6789/0 2024-03-29T10:35:09.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.051+0000 7fb9e9a14640 10 monclient(hunting): picked mon.noname-a con 0x7fb9e41457f0 addr v1:172.21.15.84:6789/0 2024-03-29T10:35:09.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.051+0000 7fb9e9a14640 1 -- 172.21.15.84:0/2055372592 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e406f610 con 0x7fb9e41457f0 2024-03-29T10:35:09.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.051+0000 7fb9e9a14640 1 -- 172.21.15.84:0/2055372592 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e4071fa0 con 0x7fb9e413f650 2024-03-29T10:35:09.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.051+0000 7fb9e9a14640 1 -- 172.21.15.84:0/2055372592 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e4136e40 con 0x7fb9e40a9d70 2024-03-29T10:35:09.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.051+0000 7fb9e9a14640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:09.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.051+0000 7fb9e9a14640 10 monclient(hunting): authenticate will time out at 811.314598s 2024-03-29T10:35:09.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.052+0000 7fb9cbfff640 10 client.?.objecter ms_handle_connect 0x7fb9e413f650 2024-03-29T10:35:09.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.052+0000 7fb9cbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:09.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.052+0000 7fb9cbfff640 1 -- 172.21.15.84:0/2055372592 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2444954572 0 0) 0x7fb9e4071fa0 con 0x7fb9e413f650 2024-03-29T10:35:09.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.052+0000 7fb9cbfff640 10 monclient(hunting): _negotiate 2024-03-29T10:35:09.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.052+0000 7fb9cbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:09.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.052+0000 7fb9cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:09.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.052+0000 7fb9cbfff640 10 monclient(hunting): my global_id is 4240 2024-03-29T10:35:09.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.052+0000 7fb9cbfff640 1 -- 172.21.15.84:0/2055372592 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9d0004eb0 con 0x7fb9e413f650 2024-03-29T10:35:09.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.053+0000 7fb9cbfff640 1 -- 172.21.15.84:0/2055372592 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1475131120 0 0) 0x7fb9d0004eb0 con 0x7fb9e413f650 2024-03-29T10:35:09.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.053+0000 7fb9cbfff640 1 -- 172.21.15.84:0/2055372592 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9e4071fa0 con 0x7fb9e413f650 2024-03-29T10:35:09.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.053+0000 7fb9cbfff640 1 -- 172.21.15.84:0/2055372592 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fb9d8001550 con 0x7fb9e413f650 2024-03-29T10:35:09.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.053+0000 7fb9cbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:09.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.053+0000 7fb9cbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:35:09.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.053+0000 7fb9cbfff640 10 monclient(hunting): dump: 2024-03-29T10:35:09.044 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:09.044 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:09.044 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:09.044 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:09.044 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:09.044 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:09.044 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:09.045 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:09.045 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:09.045 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:09.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.053+0000 7fb9cbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:35:09.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.053+0000 7fb9cbfff640 1 -- 172.21.15.84:0/2055372592 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1147683115 0 0) 0x7fb9e4071fa0 con 0x7fb9e413f650 2024-03-29T10:35:09.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.053+0000 7fb9cbfff640 1 -- 172.21.15.84:0/2055372592 >> v1:172.21.15.84:6790/0 conn(0x7fb9e40a9d70 legacy=0x7fb9e4137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:09.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.053+0000 7fb9cbfff640 1 -- 172.21.15.84:0/2055372592 >> v1:172.21.15.84:6789/0 conn(0x7fb9e41457f0 legacy=0x7fb9e406e5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:35:09.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.054+0000 7fb9cbfff640 10 monclient: _finish_hunting 0 2024-03-29T10:35:09.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.054+0000 7fb9cbfff640 1 monclient: found mon.b 2024-03-29T10:35:09.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.054+0000 7fb9cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:09.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.054+0000 7fb9cbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:35:09.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.054+0000 7fb9cbfff640 1 -- 172.21.15.84:0/2055372592 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e406f0f0 con 0x7fb9e413f650 2024-03-29T10:35:09.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.054+0000 7fb9cbfff640 10 monclient: _finish_auth 0 2024-03-29T10:35:09.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.054+0000 7fb9cbfff640 10 monclient: _check_auth_tickets 2024-03-29T10:35:09.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.054+0000 7fb9cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:09.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.054+0000 7fb9e9a14640 5 monclient: authenticate success, global_id 4240 2024-03-29T10:35:09.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.054+0000 7fb9e9a14640 10 monclient: _renew_subs 2024-03-29T10:35:09.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.054+0000 7fb9e9a14640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:35:09.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.054+0000 7fb9e9a14640 1 -- 172.21.15.84:0/2055372592 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9e406ecb0 con 0x7fb9e413f650 2024-03-29T10:35:09.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.054+0000 7fb9e9a14640 10 client.4240.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:35:09.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.054+0000 7fb9e9a14640 10 monclient: _renew_subs 2024-03-29T10:35:09.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.054+0000 7fb9e9a14640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:35:09.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.054+0000 7fb9e9a14640 1 -- 172.21.15.84:0/2055372592 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9e4136720 con 0x7fb9e413f650 2024-03-29T10:35:09.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.054+0000 7fb9e9a14640 1 librados: init done 2024-03-29T10:35:09.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.055+0000 7fb9cbfff640 1 -- 172.21.15.84:0/2055372592 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9d8004bf0 con 0x7fb9e413f650 2024-03-29T10:35:09.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.055+0000 7fb9cbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:09.047 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.055+0000 7fb9e17fa640 4 set_mon_vals no callback set 2024-03-29T10:35:09.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.056+0000 7fb9e9a14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:09.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.056+0000 7fb9cbfff640 1 -- 172.21.15.84:0/2055372592 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fb9d8004f00 con 0x7fb9e413f650 2024-03-29T10:35:09.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.056+0000 7fb9cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:09.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.056+0000 7fb9cbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:35:09.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.056+0000 7fb9cbfff640 10 monclient: dump: 2024-03-29T10:35:09.049 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:09.049 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:09.049 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:09.049 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:09.049 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:09.049 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:09.049 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:09.049 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:09.049 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:09.049 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:09.049 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.056+0000 7fb9e9a14640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:09.049 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.056+0000 7fb9e9a14640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:35:09.049 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.056+0000 7fb9e9a14640 1 -- 172.21.15.84:0/2055372592 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9c4005080 con 0x7fb9e413f650 2024-03-29T10:35:09.049 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.057+0000 7fb9cbfff640 1 -- 172.21.15.84:0/2055372592 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7fb9d801aed0 con 0x7fb9e413f650 2024-03-29T10:35:09.049 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.058+0000 7fb9e2ffd640 10 monclient: get_auth_request con 0x7fb9d006dba0 auth_method 0 2024-03-29T10:35:09.050 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.058+0000 7fb9cbfff640 1 -- 172.21.15.84:0/2055372592 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7fb9d8084c10 con 0x7fb9e413f650 2024-03-29T10:35:09.050 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.058+0000 7fb9cbfff640 10 client.4240.objecter ms_dispatch 0x7fb9e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:35:09.050 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.058+0000 7fb9cbfff640 3 client.4240.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:35:09.050 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.058+0000 7fb9cbfff640 3 client.4240.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:35:09.050 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.058+0000 7fb9cbfff640 20 client.4240.objecter dump_active .. 0 homeless 2024-03-29T10:35:09.050 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.058+0000 7fb9cbfff640 10 client.4240.objecter ms_handle_connect 0x7fb9d006dba0 2024-03-29T10:35:09.052 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.067+0000 7fb9cbfff640 1 -- 172.21.15.84:0/2055372592 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb9d80504f0 con 0x7fb9e413f650 2024-03-29T10:35:09.052 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.067+0000 7fb9cbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:09.052 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.067+0000 7fb9cbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:35:09.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.375+0000 7fb9e9a14640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:09.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.375+0000 7fb9e9a14640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:09.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.375+0000 7fb9e9a14640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:35:09.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.375+0000 7fb9e9a14640 1 -- 172.21.15.84:0/2055372592 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9c4005330 con 0x7fb9e413f650 2024-03-29T10:35:09.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.376+0000 7fb9cbfff640 1 -- 172.21.15.84:0/2055372592 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7fb9d8052420 con 0x7fb9e413f650 2024-03-29T10:35:09.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.376+0000 7fb9cbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:09.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.376+0000 7fb9cbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:35:09.362 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:35:09.362 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:35:09.365 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.379+0000 7fb9e9a14640 10 librados: watch_flush enter 2024-03-29T10:35:09.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.380+0000 7fb9e9a14640 10 librados: watch_flush exit 2024-03-29T10:35:09.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.380+0000 7fb9e9a14640 20 client.4240.objecter shutdown clearing up homeless session... 2024-03-29T10:35:09.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.380+0000 7fb9e9a14640 10 client.4240.objecter successfully canceled tick 2024-03-29T10:35:09.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.380+0000 7fb9e9a14640 1 -- 172.21.15.84:0/2055372592 >> v1:172.21.15.86:6816/3337225337 conn(0x7fb9d006dba0 legacy=0x7fb9d0070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:09.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.380+0000 7fb9e9a14640 10 monclient: shutdown 2024-03-29T10:35:09.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.380+0000 7fb9e9a14640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:09.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.380+0000 7fb9e9a14640 1 -- 172.21.15.84:0/2055372592 >> v1:172.21.15.86:6789/0 conn(0x7fb9e413f650 legacy=0x7fb9e406de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:09.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.380+0000 7fb9e9a14640 1 -- 172.21.15.84:0/2055372592 shutdown_connections 2024-03-29T10:35:09.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.380+0000 7fb9e9a14640 1 -- 172.21.15.84:0/2055372592 >> 172.21.15.84:0/2055372592 conn(0x7fb9e4066a00 msgr2=0x7fb9e40b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:09.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.381+0000 7fb9e9a14640 1 -- 172.21.15.84:0/2055372592 shutdown_connections 2024-03-29T10:35:09.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.381+0000 7fb9e9a14640 1 -- 172.21.15.84:0/2055372592 wait complete. 2024-03-29T10:35:09.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:09.381+0000 7fb9e9a14640 1 librados: shutdown 2024-03-29T10:35:09.380 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:35:16.188 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:35:16.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.357+0000 7fb46f433640 10 monclient: get_monmap_and_config 2024-03-29T10:35:16.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.357+0000 7fb46f433640 10 monclient: build_initial_monmap 2024-03-29T10:35:16.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.357+0000 7fb46f433640 10 monclient: monmap: 2024-03-29T10:35:16.343 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:35:16.344 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:16.344 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:35:16.358803+0000 2024-03-29T10:35:16.344 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:35:16.358803+0000 2024-03-29T10:35:16.344 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:16.344 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:16.344 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:35:16.344 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:35:16.344 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:35:16.344 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:16.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.358+0000 7fb46f433640 1 Processor -- start 2024-03-29T10:35:16.345 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.359+0000 7fb46f433640 1 -- start start 2024-03-29T10:35:16.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.359+0000 7fb46f433640 10 monclient: init 2024-03-29T10:35:16.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.360+0000 7fb46f433640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:16.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.360+0000 7fb46f433640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:35:16.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.360+0000 7fb46f433640 10 monclient(hunting): picked mon.noname-b con 0x7fb4680a9ba0 addr v1:172.21.15.84:6790/0 2024-03-29T10:35:16.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.360+0000 7fb46f433640 10 monclient(hunting): picked mon.noname-a con 0x7fb46813b5a0 addr v1:172.21.15.84:6789/0 2024-03-29T10:35:16.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.360+0000 7fb46f433640 10 monclient(hunting): picked mon.noname-c con 0x7fb4680ab550 addr v1:172.21.15.86:6789/0 2024-03-29T10:35:16.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.360+0000 7fb46f433640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4680ac430 con 0x7fb46813b5a0 2024-03-29T10:35:16.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.360+0000 7fb46f433640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4680ac680 con 0x7fb4680ab550 2024-03-29T10:35:16.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.360+0000 7fb46f433640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4680ac960 con 0x7fb4680a9ba0 2024-03-29T10:35:16.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.360+0000 7fb46f433640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:16.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.360+0000 7fb46f433640 10 monclient(hunting): authenticate will time out at 818.623379s 2024-03-29T10:35:16.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.362+0000 7fb46d9d0640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fb4680ab550 0x7fb4681498d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:60250/0 (socket says 172.21.15.84:60250) 2024-03-29T10:35:16.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.362+0000 7fb46d9d0640 1 -- 172.21.15.84:0/406760738 learned_addr learned my addr 172.21.15.84:0/406760738 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:35:16.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.363+0000 7fb467fff640 1 -- 172.21.15.84:0/406760738 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1982581921 0 0) 0x7fb4680ac680 con 0x7fb4680ab550 2024-03-29T10:35:16.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.363+0000 7fb467fff640 10 monclient(hunting): _negotiate 2024-03-29T10:35:16.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.363+0000 7fb467fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:16.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.363+0000 7fb467fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:16.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.363+0000 7fb467fff640 10 monclient(hunting): my global_id is 4243 2024-03-29T10:35:16.348 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.363+0000 7fb467fff640 1 -- 172.21.15.84:0/406760738 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb450003520 con 0x7fb4680ab550 2024-03-29T10:35:16.351 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.363+0000 7fb467fff640 1 -- 172.21.15.84:0/406760738 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2036433024 0 0) 0x7fb4680ac960 con 0x7fb4680a9ba0 2024-03-29T10:35:16.351 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.363+0000 7fb467fff640 10 monclient(hunting): _negotiate 2024-03-29T10:35:16.351 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.363+0000 7fb467fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:16.351 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.363+0000 7fb467fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:16.351 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.363+0000 7fb467fff640 10 monclient(hunting): my global_id is 4244 2024-03-29T10:35:16.351 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.363+0000 7fb467fff640 1 -- 172.21.15.84:0/406760738 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4680ac680 con 0x7fb4680a9ba0 2024-03-29T10:35:16.351 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.363+0000 7fb467fff640 1 -- 172.21.15.84:0/406760738 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 497689288 0 0) 0x7fb450003520 con 0x7fb4680ab550 2024-03-29T10:35:16.351 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.363+0000 7fb467fff640 1 -- 172.21.15.84:0/406760738 >> v1:172.21.15.84:6790/0 conn(0x7fb4680a9ba0 legacy=0x7fb46813ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:16.351 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.363+0000 7fb467fff640 1 -- 172.21.15.84:0/406760738 >> v1:172.21.15.84:6789/0 conn(0x7fb46813b5a0 legacy=0x7fb4680aae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:16.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.364+0000 7fb467fff640 10 monclient: _finish_hunting 0 2024-03-29T10:35:16.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.364+0000 7fb467fff640 1 monclient: found mon.noname-c 2024-03-29T10:35:16.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.364+0000 7fb467fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:16.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.364+0000 7fb467fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-29T10:35:16.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.364+0000 7fb467fff640 1 -- 172.21.15.84:0/406760738 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb46813baf0 con 0x7fb4680ab550 2024-03-29T10:35:16.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.364+0000 7fb467fff640 10 monclient: _finish_auth 0 2024-03-29T10:35:16.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.364+0000 7fb467fff640 10 monclient: _check_auth_tickets 2024-03-29T10:35:16.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.364+0000 7fb467fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:16.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.364+0000 7fb46f433640 5 monclient: authenticate success, global_id 4243 2024-03-29T10:35:16.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.364+0000 7fb46f433640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:35:16.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.364+0000 7fb467fff640 1 -- 172.21.15.84:0/406760738 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fb460001610 con 0x7fb4680ab550 2024-03-29T10:35:16.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.364+0000 7fb467fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:16.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.364+0000 7fb467fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:35:16.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.364+0000 7fb467fff640 10 monclient: dump: 2024-03-29T10:35:16.352 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:16.353 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:16.353 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:16.353 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:16.353 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:16.353 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:16.353 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:16.353 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:16.353 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:16.353 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:16.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.364+0000 7fb46f433640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:35:16.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.367+0000 7fb467fff640 1 -- 172.21.15.84:0/406760738 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb460001830 con 0x7fb4680ab550 2024-03-29T10:35:16.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.367+0000 7fb467fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:16.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.367+0000 7fb46f433640 10 monclient: get_monmap_and_config success 2024-03-29T10:35:16.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.367+0000 7fb46f433640 4 set_mon_vals no callback set 2024-03-29T10:35:16.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.367+0000 7fb46f433640 10 monclient: shutdown 2024-03-29T10:35:16.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.368+0000 7fb46f433640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:16.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.368+0000 7fb46f433640 1 -- 172.21.15.84:0/406760738 >> v1:172.21.15.86:6789/0 conn(0x7fb4680ab550 legacy=0x7fb4681498d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:16.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.368+0000 7fb46f433640 1 -- 172.21.15.84:0/406760738 shutdown_connections 2024-03-29T10:35:16.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.368+0000 7fb46f433640 1 -- 172.21.15.84:0/406760738 >> 172.21.15.84:0/406760738 conn(0x7fb4680b3a00 msgr2=0x7fb4680b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:16.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.368+0000 7fb46f433640 1 -- 172.21.15.84:0/406760738 shutdown_connections 2024-03-29T10:35:16.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.369+0000 7fb46f433640 1 -- 172.21.15.84:0/406760738 wait complete. 2024-03-29T10:35:16.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.370+0000 7fb46f433640 10 monclient: build_initial_monmap 2024-03-29T10:35:16.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.370+0000 7fb46f433640 10 monclient: monmap: 2024-03-29T10:35:16.357 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:35:16.358 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:35:16.358 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:35:16.358 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:35:16.358 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:16.358 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:16.358 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:35:16.358 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:35:16.358 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:35:16.358 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:16.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.370+0000 7fb46f433640 1 librados: starting msgr at 2024-03-29T10:35:16.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.370+0000 7fb46f433640 1 librados: starting objecter 2024-03-29T10:35:16.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.370+0000 7fb46f433640 1 Processor -- start 2024-03-29T10:35:16.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.371+0000 7fb46f433640 1 -- start start 2024-03-29T10:35:16.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.371+0000 7fb46f433640 1 librados: setting wanted keys 2024-03-29T10:35:16.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.371+0000 7fb46f433640 1 librados: calling monclient init 2024-03-29T10:35:16.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.371+0000 7fb46f433640 10 monclient: init 2024-03-29T10:35:16.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.372+0000 7fb46f433640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:16.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.372+0000 7fb46f433640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:35:16.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.372+0000 7fb46f433640 10 monclient(hunting): picked mon.noname-a con 0x7fb4680a9ba0 addr v1:172.21.15.84:6789/0 2024-03-29T10:35:16.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.372+0000 7fb46f433640 10 monclient(hunting): picked mon.noname-b con 0x7fb4680ab550 addr v1:172.21.15.86:6789/0 2024-03-29T10:35:16.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.372+0000 7fb46f433640 10 monclient(hunting): picked mon.noname-c con 0x7fb46813b5a0 addr v1:172.21.15.84:6790/0 2024-03-29T10:35:16.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.372+0000 7fb46f433640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb468140a20 con 0x7fb4680a9ba0 2024-03-29T10:35:16.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.372+0000 7fb46f433640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb468140c90 con 0x7fb4680ab550 2024-03-29T10:35:16.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.372+0000 7fb46f433640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4681e3d60 con 0x7fb46813b5a0 2024-03-29T10:35:16.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.372+0000 7fb46f433640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:16.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.372+0000 7fb46f433640 10 monclient(hunting): authenticate will time out at 818.635308s 2024-03-29T10:35:16.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.372+0000 7fb46c9ce640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fb4680ab550 0x7fb468141880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:60528/0 (socket says 172.21.15.84:60528) 2024-03-29T10:35:16.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.373+0000 7fb46c9ce640 1 -- 172.21.15.84:0/1534920137 learned_addr learned my addr 172.21.15.84:0/1534920137 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:35:16.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.373+0000 7fb465ffb640 10 client.?.objecter ms_handle_connect 0x7fb4680ab550 2024-03-29T10:35:16.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.373+0000 7fb465ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:16.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.373+0000 7fb465ffb640 1 -- 172.21.15.84:0/1534920137 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3456565185 0 0) 0x7fb468140c90 con 0x7fb4680ab550 2024-03-29T10:35:16.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.373+0000 7fb465ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:35:16.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.373+0000 7fb465ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:16.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.373+0000 7fb465ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:16.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.373+0000 7fb465ffb640 10 monclient(hunting): my global_id is 4246 2024-03-29T10:35:16.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.373+0000 7fb465ffb640 1 -- 172.21.15.84:0/1534920137 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb454004eb0 con 0x7fb4680ab550 2024-03-29T10:35:16.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.375+0000 7fb465ffb640 1 -- 172.21.15.84:0/1534920137 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 435456075 0 0) 0x7fb454004eb0 con 0x7fb4680ab550 2024-03-29T10:35:16.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.375+0000 7fb465ffb640 1 -- 172.21.15.84:0/1534920137 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb454005300 con 0x7fb4680ab550 2024-03-29T10:35:16.367 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.375+0000 7fb465ffb640 1 -- 172.21.15.84:0/1534920137 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fb458001cd0 con 0x7fb4680ab550 2024-03-29T10:35:16.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.375+0000 7fb465ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:16.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.375+0000 7fb465ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:35:16.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.375+0000 7fb465ffb640 10 monclient(hunting): dump: 2024-03-29T10:35:16.368 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:16.368 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:16.368 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:16.368 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:16.368 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:16.368 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:16.368 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:16.368 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:16.368 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:16.368 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:16.368 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.375+0000 7fb465ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:35:16.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.375+0000 7fb465ffb640 10 client.?.objecter ms_handle_connect 0x7fb46813b5a0 2024-03-29T10:35:16.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.375+0000 7fb465ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:16.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.375+0000 7fb465ffb640 1 -- 172.21.15.84:0/1534920137 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553684461 0 0) 0x7fb4681e3d60 con 0x7fb46813b5a0 2024-03-29T10:35:16.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.375+0000 7fb465ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:35:16.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.375+0000 7fb465ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:16.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.375+0000 7fb465ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:16.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.375+0000 7fb465ffb640 10 monclient(hunting): my global_id is 4247 2024-03-29T10:35:16.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.375+0000 7fb465ffb640 1 -- 172.21.15.84:0/1534920137 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4540058a0 con 0x7fb46813b5a0 2024-03-29T10:35:16.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.376+0000 7fb465ffb640 1 -- 172.21.15.84:0/1534920137 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2381563385 0 0) 0x7fb454005300 con 0x7fb4680ab550 2024-03-29T10:35:16.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.376+0000 7fb465ffb640 1 -- 172.21.15.84:0/1534920137 >> v1:172.21.15.84:6790/0 conn(0x7fb46813b5a0 legacy=0x7fb4681ef400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:16.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.376+0000 7fb465ffb640 1 -- 172.21.15.84:0/1534920137 >> v1:172.21.15.84:6789/0 conn(0x7fb4680a9ba0 legacy=0x7fb4681e4280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:35:16.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.376+0000 7fb465ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:35:16.369 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.376+0000 7fb465ffb640 1 monclient: found mon.b 2024-03-29T10:35:16.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.376+0000 7fb465ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:16.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.376+0000 7fb465ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:35:16.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.376+0000 7fb465ffb640 1 -- 172.21.15.84:0/1534920137 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4681e4990 con 0x7fb4680ab550 2024-03-29T10:35:16.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.376+0000 7fb465ffb640 10 monclient: _finish_auth 0 2024-03-29T10:35:16.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.376+0000 7fb465ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:35:16.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.376+0000 7fb465ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:16.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.376+0000 7fb46f433640 5 monclient: authenticate success, global_id 4246 2024-03-29T10:35:16.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.376+0000 7fb46f433640 10 monclient: _renew_subs 2024-03-29T10:35:16.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.376+0000 7fb46f433640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:35:16.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.376+0000 7fb46f433640 1 -- 172.21.15.84:0/1534920137 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4681410b0 con 0x7fb4680ab550 2024-03-29T10:35:16.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.376+0000 7fb46f433640 10 client.4246.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:35:16.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.376+0000 7fb46f433640 10 monclient: _renew_subs 2024-03-29T10:35:16.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.376+0000 7fb46f433640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:35:16.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.376+0000 7fb46f433640 1 -- 172.21.15.84:0/1534920137 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb468141f90 con 0x7fb4680ab550 2024-03-29T10:35:16.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.376+0000 7fb46f433640 1 librados: init done 2024-03-29T10:35:16.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.377+0000 7fb465ffb640 1 -- 172.21.15.84:0/1534920137 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb458001fd0 con 0x7fb4680ab550 2024-03-29T10:35:16.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.377+0000 7fb465ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:16.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.377+0000 7fb4677fe640 4 set_mon_vals no callback set 2024-03-29T10:35:16.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.377+0000 7fb465ffb640 1 -- 172.21.15.84:0/1534920137 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fb458001cd0 con 0x7fb4680ab550 2024-03-29T10:35:16.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.377+0000 7fb465ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:16.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.377+0000 7fb465ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:35:16.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.377+0000 7fb465ffb640 10 monclient: dump: 2024-03-29T10:35:16.373 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:16.373 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:16.373 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:16.373 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:16.373 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:16.373 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:16.373 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:16.373 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:16.373 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:16.374 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:16.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.378+0000 7fb46f433640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:16.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.378+0000 7fb46f433640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:16.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.378+0000 7fb46f433640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:35:16.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.378+0000 7fb46f433640 1 -- 172.21.15.84:0/1534920137 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb448005080 con 0x7fb4680ab550 2024-03-29T10:35:16.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.381+0000 7fb465ffb640 1 -- 172.21.15.84:0/1534920137 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7fb458003cc0 con 0x7fb4680ab550 2024-03-29T10:35:16.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.382+0000 7fb465ffb640 1 -- 172.21.15.84:0/1534920137 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7fb458084bd0 con 0x7fb4680ab550 2024-03-29T10:35:16.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.382+0000 7fb465ffb640 10 client.4246.objecter ms_dispatch 0x7fb468000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:35:16.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.382+0000 7fb465ffb640 3 client.4246.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:35:16.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.382+0000 7fb465ffb640 3 client.4246.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:35:16.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.382+0000 7fb465ffb640 20 client.4246.objecter dump_active .. 0 homeless 2024-03-29T10:35:16.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.382+0000 7fb46d1cf640 10 monclient: get_auth_request con 0x7fb45406e330 auth_method 0 2024-03-29T10:35:16.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.383+0000 7fb465ffb640 10 client.4246.objecter ms_handle_connect 0x7fb45406e330 2024-03-29T10:35:16.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.386+0000 7fb465ffb640 1 -- 172.21.15.84:0/1534920137 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb4580504b0 con 0x7fb4680ab550 2024-03-29T10:35:16.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.386+0000 7fb465ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:16.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.386+0000 7fb465ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:35:16.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.707+0000 7fb46f433640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:16.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.707+0000 7fb46f433640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:16.693 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.707+0000 7fb46f433640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:35:16.693 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.707+0000 7fb46f433640 1 -- 172.21.15.84:0/1534920137 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb448005330 con 0x7fb4680ab550 2024-03-29T10:35:16.693 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.709+0000 7fb465ffb640 1 -- 172.21.15.84:0/1534920137 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7fb4580523e0 con 0x7fb4680ab550 2024-03-29T10:35:16.693 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.709+0000 7fb465ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:16.694 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.709+0000 7fb465ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:35:16.696 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:35:16.697 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:35:16.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.715+0000 7fb46f433640 10 librados: watch_flush enter 2024-03-29T10:35:16.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.715+0000 7fb46f433640 10 librados: watch_flush exit 2024-03-29T10:35:16.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.715+0000 7fb46f433640 20 client.4246.objecter shutdown clearing up homeless session... 2024-03-29T10:35:16.701 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.715+0000 7fb46f433640 10 client.4246.objecter successfully canceled tick 2024-03-29T10:35:16.701 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.715+0000 7fb46f433640 1 -- 172.21.15.84:0/1534920137 >> v1:172.21.15.86:6816/3337225337 conn(0x7fb45406e330 legacy=0x7fb4540707f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:16.701 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.715+0000 7fb46f433640 10 monclient: shutdown 2024-03-29T10:35:16.701 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.715+0000 7fb46f433640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:16.701 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.715+0000 7fb46f433640 1 -- 172.21.15.84:0/1534920137 >> v1:172.21.15.86:6789/0 conn(0x7fb4680ab550 legacy=0x7fb468141880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:16.702 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.717+0000 7fb46f433640 1 -- 172.21.15.84:0/1534920137 shutdown_connections 2024-03-29T10:35:16.702 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.717+0000 7fb46f433640 1 -- 172.21.15.84:0/1534920137 >> 172.21.15.84:0/1534920137 conn(0x7fb4680b3a00 msgr2=0x7fb4680b0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:16.702 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.717+0000 7fb46f433640 1 -- 172.21.15.84:0/1534920137 shutdown_connections 2024-03-29T10:35:16.702 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.717+0000 7fb46f433640 1 -- 172.21.15.84:0/1534920137 wait complete. 2024-03-29T10:35:16.702 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:16.717+0000 7fb46f433640 1 librados: shutdown 2024-03-29T10:35:16.719 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:35:23.526 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:35:23.669 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.684+0000 7f0f16dab640 10 monclient: get_monmap_and_config 2024-03-29T10:35:23.670 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.684+0000 7f0f16dab640 10 monclient: build_initial_monmap 2024-03-29T10:35:23.670 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.684+0000 7f0f16dab640 10 monclient: monmap: 2024-03-29T10:35:23.670 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:35:23.670 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:23.670 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:35:23.685571+0000 2024-03-29T10:35:23.670 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:35:23.685571+0000 2024-03-29T10:35:23.670 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:23.670 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:23.670 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:35:23.670 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:35:23.670 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:35:23.671 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:23.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.685+0000 7f0f16dab640 1 Processor -- start 2024-03-29T10:35:23.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.685+0000 7f0f16dab640 1 -- start start 2024-03-29T10:35:23.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.685+0000 7f0f16dab640 10 monclient: init 2024-03-29T10:35:23.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.686+0000 7f0f16dab640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:23.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.686+0000 7f0f16dab640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:35:23.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.686+0000 7f0f16dab640 10 monclient(hunting): picked mon.noname-b con 0x7f0f100b1f60 addr v1:172.21.15.84:6790/0 2024-03-29T10:35:23.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.686+0000 7f0f16dab640 10 monclient(hunting): picked mon.noname-a con 0x7f0f10071eb0 addr v1:172.21.15.84:6789/0 2024-03-29T10:35:23.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.686+0000 7f0f16dab640 10 monclient(hunting): picked mon.noname-c con 0x7f0f10143d10 addr v1:172.21.15.86:6789/0 2024-03-29T10:35:23.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.686+0000 7f0f16dab640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1006ef80 con 0x7f0f10071eb0 2024-03-29T10:35:23.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.686+0000 7f0f16dab640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1006f260 con 0x7f0f10143d10 2024-03-29T10:35:23.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.686+0000 7f0f16dab640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f100b40c0 con 0x7f0f100b1f60 2024-03-29T10:35:23.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.686+0000 7f0f16dab640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:23.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.686+0000 7f0f16dab640 10 monclient(hunting): authenticate will time out at 825.949689s 2024-03-29T10:35:23.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.689+0000 7f0f14b47640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f0f100b1f60 0x7f0f10143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:43259/0 (socket says 172.21.15.84:43259) 2024-03-29T10:35:23.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.689+0000 7f0f14b47640 1 -- 172.21.15.84:0/3131730826 learned_addr learned my addr 172.21.15.84:0/3131730826 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:35:23.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.689+0000 7f0f0f7fe640 1 -- 172.21.15.84:0/3131730826 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 525880487 0 0) 0x7f0f100b40c0 con 0x7f0f100b1f60 2024-03-29T10:35:23.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.689+0000 7f0f0f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:35:23.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.689+0000 7f0f0f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:23.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.689+0000 7f0f0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:23.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.689+0000 7f0f0f7fe640 10 monclient(hunting): my global_id is 4250 2024-03-29T10:35:23.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.689+0000 7f0f0f7fe640 1 -- 172.21.15.84:0/3131730826 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f00003520 con 0x7f0f100b1f60 2024-03-29T10:35:23.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.689+0000 7f0f0f7fe640 1 -- 172.21.15.84:0/3131730826 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2088105386 0 0) 0x7f0f00003520 con 0x7f0f100b1f60 2024-03-29T10:35:23.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.689+0000 7f0f0f7fe640 1 -- 172.21.15.84:0/3131730826 >> v1:172.21.15.86:6789/0 conn(0x7f0f10143d10 legacy=0x7f0f10146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:35:23.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.689+0000 7f0f0f7fe640 1 -- 172.21.15.84:0/3131730826 >> v1:172.21.15.84:6789/0 conn(0x7f0f10071eb0 legacy=0x7f0f10146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:35:23.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.690+0000 7f0f0f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:35:23.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.690+0000 7f0f0f7fe640 1 monclient: found mon.noname-b 2024-03-29T10:35:23.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.690+0000 7f0f0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:23.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.690+0000 7f0f0f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:35:23.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.690+0000 7f0f0f7fe640 1 -- 172.21.15.84:0/3131730826 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f10155d70 con 0x7f0f100b1f60 2024-03-29T10:35:23.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.690+0000 7f0f0f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:35:23.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.690+0000 7f0f0f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:35:23.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.690+0000 7f0f0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:23.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.690+0000 7f0f16dab640 5 monclient: authenticate success, global_id 4250 2024-03-29T10:35:23.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.690+0000 7f0f0f7fe640 1 -- 172.21.15.84:0/3131730826 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0f04001690 con 0x7f0f100b1f60 2024-03-29T10:35:23.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.690+0000 7f0f16dab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:35:23.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.690+0000 7f0f0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:23.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.690+0000 7f0f0f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:35:23.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.690+0000 7f0f0f7fe640 10 monclient: dump: 2024-03-29T10:35:23.677 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:23.677 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:23.677 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:23.677 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:23.677 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:23.677 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:23.677 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:23.678 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:23.678 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:23.678 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:23.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.690+0000 7f0f16dab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:35:23.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.690+0000 7f0f0f7fe640 1 -- 172.21.15.84:0/3131730826 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f040018f0 con 0x7f0f100b1f60 2024-03-29T10:35:23.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.690+0000 7f0f0f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:23.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.690+0000 7f0f16dab640 10 monclient: get_monmap_and_config success 2024-03-29T10:35:23.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.690+0000 7f0f16dab640 4 set_mon_vals no callback set 2024-03-29T10:35:23.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.690+0000 7f0f0f7fe640 1 -- 172.21.15.84:0/3131730826 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0f04001d40 con 0x7f0f100b1f60 2024-03-29T10:35:23.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.690+0000 7f0f0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:23.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.690+0000 7f0f0f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:35:23.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.690+0000 7f0f0f7fe640 10 monclient: dump: 2024-03-29T10:35:23.678 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:23.678 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:23.679 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:23.679 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:23.679 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:23.679 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:23.679 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:23.679 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:23.679 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:23.679 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:23.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.691+0000 7f0f16dab640 10 monclient: shutdown 2024-03-29T10:35:23.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.691+0000 7f0f16dab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:23.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.691+0000 7f0f16dab640 1 -- 172.21.15.84:0/3131730826 >> v1:172.21.15.84:6790/0 conn(0x7f0f100b1f60 legacy=0x7f0f10143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:23.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.691+0000 7f0f16dab640 1 -- 172.21.15.84:0/3131730826 shutdown_connections 2024-03-29T10:35:23.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.691+0000 7f0f16dab640 1 -- 172.21.15.84:0/3131730826 >> 172.21.15.84:0/3131730826 conn(0x7f0f10066a00 msgr2=0x7f0f10066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:23.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.691+0000 7f0f16dab640 1 -- 172.21.15.84:0/3131730826 shutdown_connections 2024-03-29T10:35:23.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.691+0000 7f0f16dab640 1 -- 172.21.15.84:0/3131730826 wait complete. 2024-03-29T10:35:23.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.692+0000 7f0f16dab640 10 monclient: build_initial_monmap 2024-03-29T10:35:23.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.692+0000 7f0f16dab640 10 monclient: monmap: 2024-03-29T10:35:23.681 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:35:23.681 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:35:23.681 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:35:23.682 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:35:23.682 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:23.682 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:23.682 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:35:23.682 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:35:23.682 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:35:23.682 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:23.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.693+0000 7f0f16dab640 1 librados: starting msgr at 2024-03-29T10:35:23.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.693+0000 7f0f16dab640 1 librados: starting objecter 2024-03-29T10:35:23.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.693+0000 7f0f16dab640 1 Processor -- start 2024-03-29T10:35:23.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.693+0000 7f0f16dab640 1 -- start start 2024-03-29T10:35:23.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.693+0000 7f0f16dab640 1 librados: setting wanted keys 2024-03-29T10:35:23.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.693+0000 7f0f16dab640 1 librados: calling monclient init 2024-03-29T10:35:23.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.693+0000 7f0f16dab640 10 monclient: init 2024-03-29T10:35:23.683 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.694+0000 7f0f16dab640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:23.683 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.694+0000 7f0f16dab640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:35:23.683 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.694+0000 7f0f16dab640 10 monclient(hunting): picked mon.noname-c con 0x7f0f100b1f60 addr v1:172.21.15.84:6790/0 2024-03-29T10:35:23.683 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.694+0000 7f0f16dab640 10 monclient(hunting): picked mon.noname-a con 0x7f0f10143d10 addr v1:172.21.15.84:6789/0 2024-03-29T10:35:23.683 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.694+0000 7f0f16dab640 10 monclient(hunting): picked mon.noname-b con 0x7f0f101546a0 addr v1:172.21.15.86:6789/0 2024-03-29T10:35:23.683 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.694+0000 7f0f16dab640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f10155580 con 0x7f0f10143d10 2024-03-29T10:35:23.683 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.694+0000 7f0f16dab640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f100708f0 con 0x7f0f101546a0 2024-03-29T10:35:23.683 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.694+0000 7f0f16dab640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f10070bd0 con 0x7f0f100b1f60 2024-03-29T10:35:23.683 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.694+0000 7f0f16dab640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:23.683 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.694+0000 7f0f16dab640 10 monclient(hunting): authenticate will time out at 825.957500s 2024-03-29T10:35:23.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.698+0000 7f0f14b47640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f0f100b1f60 0x7f0f10070ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:43323/0 (socket says 172.21.15.84:43323) 2024-03-29T10:35:23.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.698+0000 7f0f14b47640 1 -- 172.21.15.84:0/1161856475 learned_addr learned my addr 172.21.15.84:0/1161856475 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:35:23.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.698+0000 7f0f0d7fa640 10 client.?.objecter ms_handle_connect 0x7f0f100b1f60 2024-03-29T10:35:23.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.698+0000 7f0f0d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:23.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.698+0000 7f0f0d7fa640 1 -- 172.21.15.84:0/1161856475 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 156750724 0 0) 0x7f0f10070bd0 con 0x7f0f100b1f60 2024-03-29T10:35:23.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.698+0000 7f0f0d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:35:23.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.698+0000 7f0f0d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:23.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.698+0000 7f0f0d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:23.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.698+0000 7f0f0d7fa640 10 monclient(hunting): my global_id is 4253 2024-03-29T10:35:23.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.698+0000 7f0f0d7fa640 1 -- 172.21.15.84:0/1161856475 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ef4004eb0 con 0x7f0f100b1f60 2024-03-29T10:35:23.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.699+0000 7f0f0d7fa640 10 client.?.objecter ms_handle_connect 0x7f0f10143d10 2024-03-29T10:35:23.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.699+0000 7f0f0d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:23.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.699+0000 7f0f0d7fa640 1 -- 172.21.15.84:0/1161856475 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2628107201 0 0) 0x7f0f10155580 con 0x7f0f10143d10 2024-03-29T10:35:23.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.699+0000 7f0f0d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:35:23.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.699+0000 7f0f0d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:23.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.699+0000 7f0f0d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:23.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.699+0000 7f0f0d7fa640 10 monclient(hunting): my global_id is 4251 2024-03-29T10:35:23.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.699+0000 7f0f0d7fa640 1 -- 172.21.15.84:0/1161856475 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ef40051d0 con 0x7f0f10143d10 2024-03-29T10:35:23.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.699+0000 7f0f0d7fa640 1 -- 172.21.15.84:0/1161856475 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 638135609 0 0) 0x7f0ef40051d0 con 0x7f0f10143d10 2024-03-29T10:35:23.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.699+0000 7f0f0d7fa640 1 -- 172.21.15.84:0/1161856475 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ef4005640 con 0x7f0f10143d10 2024-03-29T10:35:23.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.699+0000 7f0f0d7fa640 1 -- 172.21.15.84:0/1161856475 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0efc001810 con 0x7f0f10143d10 2024-03-29T10:35:23.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.699+0000 7f0f0d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:23.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.699+0000 7f0f0d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:35:23.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.699+0000 7f0f0d7fa640 10 monclient(hunting): dump: 2024-03-29T10:35:23.686 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:23.686 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:23.686 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:23.686 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:23.686 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:23.686 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:23.687 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:23.687 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:23.687 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:23.687 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:23.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.699+0000 7f0f0d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:35:23.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.699+0000 7f0f0d7fa640 1 -- 172.21.15.84:0/1161856475 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2156890992 0 0) 0x7f0ef4004eb0 con 0x7f0f100b1f60 2024-03-29T10:35:23.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.699+0000 7f0f0d7fa640 1 -- 172.21.15.84:0/1161856475 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ef4005d30 con 0x7f0f100b1f60 2024-03-29T10:35:23.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.699+0000 7f0f0d7fa640 1 -- 172.21.15.84:0/1161856475 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0f040014b0 con 0x7f0f100b1f60 2024-03-29T10:35:23.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.699+0000 7f0f0d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:23.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.699+0000 7f0f0d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:35:23.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.699+0000 7f0f0d7fa640 10 monclient(hunting): dump: 2024-03-29T10:35:23.687 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:23.687 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:23.687 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:23.688 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:23.688 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:23.688 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:23.688 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:23.688 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:23.688 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:23.688 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:23.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.699+0000 7f0f0d7fa640 1 -- 172.21.15.84:0/1161856475 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4112139961 0 0) 0x7f0ef4005640 con 0x7f0f10143d10 2024-03-29T10:35:23.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.699+0000 7f0f0d7fa640 1 -- 172.21.15.84:0/1161856475 >> v1:172.21.15.84:6790/0 conn(0x7f0f100b1f60 legacy=0x7f0f10070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:23.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.699+0000 7f0f0d7fa640 1 -- 172.21.15.84:0/1161856475 >> v1:172.21.15.86:6789/0 conn(0x7f0f101546a0 legacy=0x7f0f101e6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:23.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.699+0000 7f0f0d7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:35:23.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.699+0000 7f0f0d7fa640 1 monclient: found mon.a 2024-03-29T10:35:23.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.700+0000 7f0f0d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:23.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.700+0000 7f0f0d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:35:23.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.700+0000 7f0f0d7fa640 1 -- 172.21.15.84:0/1161856475 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f10155270 con 0x7f0f10143d10 2024-03-29T10:35:23.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.700+0000 7f0f0d7fa640 10 monclient: _finish_auth 0 2024-03-29T10:35:23.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.700+0000 7f0f0d7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:35:23.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.700+0000 7f0f0d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:23.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.700+0000 7f0f16dab640 5 monclient: authenticate success, global_id 4251 2024-03-29T10:35:23.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.700+0000 7f0f16dab640 10 monclient: _renew_subs 2024-03-29T10:35:23.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.700+0000 7f0f16dab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:35:23.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.700+0000 7f0f16dab640 1 -- 172.21.15.84:0/1161856475 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f10153500 con 0x7f0f10143d10 2024-03-29T10:35:23.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.700+0000 7f0f16dab640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:35:23.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.700+0000 7f0f16dab640 10 monclient: _renew_subs 2024-03-29T10:35:23.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.700+0000 7f0f16dab640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:35:23.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.700+0000 7f0f16dab640 1 -- 172.21.15.84:0/1161856475 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f1006f970 con 0x7f0f10143d10 2024-03-29T10:35:23.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.700+0000 7f0f16dab640 1 librados: init done 2024-03-29T10:35:23.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.701+0000 7f0efaffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:23.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.701+0000 7f0efaffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:23.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.701+0000 7f0efaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:35:23.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.701+0000 7f0efaffd640 1 -- 172.21.15.84:0/1161856475 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ef0005080 con 0x7f0f10143d10 2024-03-29T10:35:23.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.702+0000 7f0f0d7fa640 1 -- 172.21.15.84:0/1161856475 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0efc001d00 con 0x7f0f10143d10 2024-03-29T10:35:23.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.702+0000 7f0f0d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:23.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.702+0000 7f0f0effd640 4 set_mon_vals no callback set 2024-03-29T10:35:23.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.703+0000 7f0f0d7fa640 1 -- 172.21.15.84:0/1161856475 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0efc001810 con 0x7f0f10143d10 2024-03-29T10:35:23.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.703+0000 7f0f0d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:23.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.703+0000 7f0f0d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:35:23.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.703+0000 7f0f0d7fa640 10 monclient: dump: 2024-03-29T10:35:23.692 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:23.692 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:23.692 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:23.692 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:23.692 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:23.692 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:23.693 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:23.693 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:23.693 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:23.693 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:23.693 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.703+0000 7f0f0d7fa640 1 -- 172.21.15.84:0/1161856475 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f0efc019560 con 0x7f0f10143d10 2024-03-29T10:35:23.693 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.704+0000 7f0f14b47640 10 monclient: get_auth_request con 0x7f0ef406e8a0 auth_method 0 2024-03-29T10:35:23.693 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.704+0000 7f0f0d7fa640 1 -- 172.21.15.84:0/1161856475 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f0efc083300 con 0x7f0f10143d10 2024-03-29T10:35:23.693 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.704+0000 7f0f0d7fa640 10 client.4251.objecter ms_dispatch 0x7f0f10000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:35:23.693 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.704+0000 7f0f0d7fa640 3 client.4251.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:35:23.693 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.705+0000 7f0f0d7fa640 3 client.4251.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:35:23.693 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.705+0000 7f0f0d7fa640 20 client.4251.objecter dump_active .. 0 homeless 2024-03-29T10:35:23.693 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.705+0000 7f0f0d7fa640 10 client.4251.objecter ms_handle_connect 0x7f0ef406e8a0 2024-03-29T10:35:23.695 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.710+0000 7f0f0d7fa640 1 -- 172.21.15.84:0/1161856475 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0efc04ec10 con 0x7f0f10143d10 2024-03-29T10:35:23.695 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.710+0000 7f0f0d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:23.695 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:23.710+0000 7f0f0d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:35:24.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:24.035+0000 7f0efaffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:24.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:24.035+0000 7f0efaffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:24.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:24.035+0000 7f0efaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:35:24.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:24.035+0000 7f0efaffd640 1 -- 172.21.15.84:0/1161856475 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ef0005330 con 0x7f0f10143d10 2024-03-29T10:35:24.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:24.037+0000 7f0f0d7fa640 1 -- 172.21.15.84:0/1161856475 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f0efc04ec10 con 0x7f0f10143d10 2024-03-29T10:35:24.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:24.037+0000 7f0f0d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:24.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:24.037+0000 7f0f0d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:35:24.023 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:35:24.023 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:35:24.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:24.041+0000 7f0efaffd640 10 librados: watch_flush enter 2024-03-29T10:35:24.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:24.041+0000 7f0efaffd640 10 librados: watch_flush exit 2024-03-29T10:35:24.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:24.041+0000 7f0efaffd640 20 client.4251.objecter shutdown clearing up homeless session... 2024-03-29T10:35:24.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:24.041+0000 7f0efaffd640 10 client.4251.objecter successfully canceled tick 2024-03-29T10:35:24.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:24.041+0000 7f0efaffd640 1 -- 172.21.15.84:0/1161856475 >> v1:172.21.15.86:6816/3337225337 conn(0x7f0ef406e8a0 legacy=0x7f0ef4070d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:24.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:24.041+0000 7f0efaffd640 10 monclient: shutdown 2024-03-29T10:35:24.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:24.041+0000 7f0efaffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:24.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:24.041+0000 7f0efaffd640 1 -- 172.21.15.84:0/1161856475 >> v1:172.21.15.84:6789/0 conn(0x7f0f10143d10 legacy=0x7f0f101539f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:24.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:24.041+0000 7f0efaffd640 1 -- 172.21.15.84:0/1161856475 shutdown_connections 2024-03-29T10:35:24.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:24.041+0000 7f0efaffd640 1 -- 172.21.15.84:0/1161856475 >> 172.21.15.84:0/1161856475 conn(0x7f0f10066a00 msgr2=0x7f0f100b4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:24.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:24.042+0000 7f0efaffd640 1 -- 172.21.15.84:0/1161856475 shutdown_connections 2024-03-29T10:35:24.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:24.042+0000 7f0efaffd640 1 -- 172.21.15.84:0/1161856475 wait complete. 2024-03-29T10:35:24.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:24.042+0000 7f0efaffd640 1 librados: shutdown 2024-03-29T10:35:24.044 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:35:30.850 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:35:31.002 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.016+0000 7f739e64e640 10 monclient: get_monmap_and_config 2024-03-29T10:35:31.002 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.017+0000 7f739e64e640 10 monclient: build_initial_monmap 2024-03-29T10:35:31.002 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.017+0000 7f739e64e640 10 monclient: monmap: 2024-03-29T10:35:31.002 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:35:31.002 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:31.003 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:35:31.017977+0000 2024-03-29T10:35:31.003 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:35:31.017977+0000 2024-03-29T10:35:31.003 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:31.003 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:31.003 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:35:31.003 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:35:31.003 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:35:31.003 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:31.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.020+0000 7f739e64e640 1 Processor -- start 2024-03-29T10:35:31.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.023+0000 7f739e64e640 1 -- start start 2024-03-29T10:35:31.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.023+0000 7f739e64e640 10 monclient: init 2024-03-29T10:35:31.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.023+0000 7f739e64e640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:31.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.023+0000 7f739e64e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:35:31.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.023+0000 7f739e64e640 10 monclient(hunting): picked mon.noname-a con 0x7f73981457f0 addr v1:172.21.15.84:6789/0 2024-03-29T10:35:31.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.023+0000 7f739e64e640 10 monclient(hunting): picked mon.noname-c con 0x7f739813f650 addr v1:172.21.15.86:6789/0 2024-03-29T10:35:31.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.023+0000 7f739e64e640 10 monclient(hunting): picked mon.noname-b con 0x7f73980a9d70 addr v1:172.21.15.84:6790/0 2024-03-29T10:35:31.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.024+0000 7f739e64e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73980aaef0 con 0x7f73981457f0 2024-03-29T10:35:31.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.024+0000 7f739e64e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7398145bd0 con 0x7f739813f650 2024-03-29T10:35:31.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.024+0000 7f739e64e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7398145eb0 con 0x7f73980a9d70 2024-03-29T10:35:31.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.024+0000 7f739e64e640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:31.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.024+0000 7f739e64e640 10 monclient(hunting): authenticate will time out at 833.287218s 2024-03-29T10:35:31.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.027+0000 7f739cbeb640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f73980a9d70 0x7f739814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:45891/0 (socket says 172.21.15.84:45891) 2024-03-29T10:35:31.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.027+0000 7f739cbeb640 1 -- 172.21.15.84:0/2089840375 learned_addr learned my addr 172.21.15.84:0/2089840375 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:35:31.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 1 -- 172.21.15.84:0/2089840375 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483211286 0 0) 0x7f7398145eb0 con 0x7f73980a9d70 2024-03-29T10:35:31.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:35:31.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:31.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:31.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 10 monclient(hunting): my global_id is 4256 2024-03-29T10:35:31.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 1 -- 172.21.15.84:0/2089840375 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7380003520 con 0x7f73980a9d70 2024-03-29T10:35:31.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 1 -- 172.21.15.84:0/2089840375 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3258424278 0 0) 0x7f7398145bd0 con 0x7f739813f650 2024-03-29T10:35:31.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:35:31.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:31.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:31.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 10 monclient(hunting): my global_id is 4249 2024-03-29T10:35:31.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 1 -- 172.21.15.84:0/2089840375 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7398145eb0 con 0x7f739813f650 2024-03-29T10:35:31.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 1 -- 172.21.15.84:0/2089840375 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4151194107 0 0) 0x7f7380003520 con 0x7f73980a9d70 2024-03-29T10:35:31.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 1 -- 172.21.15.84:0/2089840375 >> v1:172.21.15.86:6789/0 conn(0x7f739813f650 legacy=0x7f73980b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:31.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 1 -- 172.21.15.84:0/2089840375 >> v1:172.21.15.84:6789/0 conn(0x7f73981457f0 legacy=0x7f739813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:35:31.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:35:31.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 1 monclient: found mon.noname-b 2024-03-29T10:35:31.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:31.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:35:31.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 1 -- 172.21.15.84:0/2089840375 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7398146340 con 0x7f73980a9d70 2024-03-29T10:35:31.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 10 monclient: _finish_auth 0 2024-03-29T10:35:31.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:35:31.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:31.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 1 -- 172.21.15.84:0/2089840375 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f738c001690 con 0x7f73980a9d70 2024-03-29T10:35:31.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:31.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:35:31.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 10 monclient: dump: 2024-03-29T10:35:31.017 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:31.017 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:31.017 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:31.017 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:31.017 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:31.017 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:31.018 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:31.018 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:31.018 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:31.018 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:31.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 1 -- 172.21.15.84:0/2089840375 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f738c001870 con 0x7f73980a9d70 2024-03-29T10:35:31.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:31.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 1 -- 172.21.15.84:0/2089840375 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f738c001690 con 0x7f73980a9d70 2024-03-29T10:35:31.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:31.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.028+0000 7f7396ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:35:31.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.029+0000 7f7396ffd640 10 monclient: dump: 2024-03-29T10:35:31.018 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:31.019 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:31.019 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:31.019 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:31.019 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:31.019 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:31.019 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:31.019 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:31.019 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:31.019 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:31.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.030+0000 7f739e64e640 5 monclient: authenticate success, global_id 4256 2024-03-29T10:35:31.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.030+0000 7f739e64e640 10 monclient: get_monmap_and_config success 2024-03-29T10:35:31.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.030+0000 7f739e64e640 4 set_mon_vals no callback set 2024-03-29T10:35:31.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.032+0000 7f739e64e640 10 monclient: shutdown 2024-03-29T10:35:31.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.032+0000 7f739e64e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:31.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.032+0000 7f739e64e640 1 -- 172.21.15.84:0/2089840375 >> v1:172.21.15.84:6790/0 conn(0x7f73980a9d70 legacy=0x7f739814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:31.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.035+0000 7f739e64e640 1 -- 172.21.15.84:0/2089840375 shutdown_connections 2024-03-29T10:35:31.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.035+0000 7f739e64e640 1 -- 172.21.15.84:0/2089840375 >> 172.21.15.84:0/2089840375 conn(0x7f7398066a00 msgr2=0x7f7398066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:31.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.038+0000 7f739e64e640 1 -- 172.21.15.84:0/2089840375 shutdown_connections 2024-03-29T10:35:31.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.038+0000 7f739e64e640 1 -- 172.21.15.84:0/2089840375 wait complete. 2024-03-29T10:35:31.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.039+0000 7f739e64e640 10 monclient: build_initial_monmap 2024-03-29T10:35:31.032 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.039+0000 7f739e64e640 10 monclient: monmap: 2024-03-29T10:35:31.032 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:35:31.032 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:35:31.032 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:35:31.032 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:35:31.032 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:31.033 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:31.033 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:35:31.033 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:35:31.033 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:35:31.033 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:31.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.039+0000 7f739e64e640 1 librados: starting msgr at 2024-03-29T10:35:31.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.039+0000 7f739e64e640 1 librados: starting objecter 2024-03-29T10:35:31.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.039+0000 7f739e64e640 1 Processor -- start 2024-03-29T10:35:31.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.040+0000 7f739e64e640 1 -- start start 2024-03-29T10:35:31.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.040+0000 7f739e64e640 1 librados: setting wanted keys 2024-03-29T10:35:31.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.040+0000 7f739e64e640 1 librados: calling monclient init 2024-03-29T10:35:31.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.040+0000 7f739e64e640 10 monclient: init 2024-03-29T10:35:31.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.040+0000 7f739e64e640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:31.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.040+0000 7f739e64e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:35:31.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.040+0000 7f739e64e640 10 monclient(hunting): picked mon.noname-b con 0x7f73980a9d70 addr v1:172.21.15.86:6789/0 2024-03-29T10:35:31.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.040+0000 7f739e64e640 10 monclient(hunting): picked mon.noname-a con 0x7f739813f650 addr v1:172.21.15.84:6789/0 2024-03-29T10:35:31.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.040+0000 7f739e64e640 10 monclient(hunting): picked mon.noname-c con 0x7f73981457f0 addr v1:172.21.15.84:6790/0 2024-03-29T10:35:31.035 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.040+0000 7f739e64e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7398137840 con 0x7f739813f650 2024-03-29T10:35:31.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.040+0000 7f739e64e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73980ac680 con 0x7f73980a9d70 2024-03-29T10:35:31.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.040+0000 7f739e64e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73980ac8d0 con 0x7f73981457f0 2024-03-29T10:35:31.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.040+0000 7f739e64e640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:31.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.040+0000 7f739e64e640 10 monclient(hunting): authenticate will time out at 833.304093s 2024-03-29T10:35:31.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.046+0000 7f739cbeb640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f73981457f0 0x7f73981f3380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:45999/0 (socket says 172.21.15.84:45999) 2024-03-29T10:35:31.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.046+0000 7f739cbeb640 1 -- 172.21.15.84:0/1918753838 learned_addr learned my addr 172.21.15.84:0/1918753838 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:35:31.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.049+0000 7f7394ff9640 10 client.?.objecter ms_handle_connect 0x7f739813f650 2024-03-29T10:35:31.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.049+0000 7f7394ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:31.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.049+0000 7f7394ff9640 10 client.?.objecter ms_handle_connect 0x7f73981457f0 2024-03-29T10:35:31.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.049+0000 7f7394ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:31.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.049+0000 7f7394ff9640 1 -- 172.21.15.84:0/1918753838 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2565641094 0 0) 0x7f73980ac8d0 con 0x7f73981457f0 2024-03-29T10:35:31.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.049+0000 7f7394ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:35:31.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.049+0000 7f7394ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:31.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.049+0000 7f7394ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:31.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.049+0000 7f7394ff9640 10 monclient(hunting): my global_id is 4259 2024-03-29T10:35:31.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.049+0000 7f7394ff9640 1 -- 172.21.15.84:0/1918753838 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7384004eb0 con 0x7f73981457f0 2024-03-29T10:35:31.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.049+0000 7f7394ff9640 1 -- 172.21.15.84:0/1918753838 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1034996627 0 0) 0x7f7398137840 con 0x7f739813f650 2024-03-29T10:35:31.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.049+0000 7f7394ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:35:31.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.049+0000 7f7394ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:31.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.049+0000 7f7394ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:31.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.049+0000 7f7394ff9640 10 monclient(hunting): my global_id is 4254 2024-03-29T10:35:31.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.049+0000 7f7394ff9640 1 -- 172.21.15.84:0/1918753838 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73980ac8d0 con 0x7f739813f650 2024-03-29T10:35:31.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.050+0000 7f7394ff9640 10 client.?.objecter ms_handle_connect 0x7f73980a9d70 2024-03-29T10:35:31.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.050+0000 7f7394ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:31.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.050+0000 7f7394ff9640 1 -- 172.21.15.84:0/1918753838 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2308495529 0 0) 0x7f73980ac8d0 con 0x7f739813f650 2024-03-29T10:35:31.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.050+0000 7f7394ff9640 1 -- 172.21.15.84:0/1918753838 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7398137840 con 0x7f739813f650 2024-03-29T10:35:31.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.050+0000 7f7394ff9640 1 -- 172.21.15.84:0/1918753838 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f7390001c30 con 0x7f739813f650 2024-03-29T10:35:31.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.050+0000 7f7394ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:31.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.050+0000 7f7394ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:35:31.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.050+0000 7f7394ff9640 10 monclient(hunting): dump: 2024-03-29T10:35:31.039 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:31.039 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:31.039 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:31.040 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:31.040 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:31.040 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:31.040 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:31.040 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:31.040 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:31.040 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:31.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.050+0000 7f7394ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:35:31.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.053+0000 7f7394ff9640 1 -- 172.21.15.84:0/1918753838 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2805065231 0 0) 0x7f7384004eb0 con 0x7f73981457f0 2024-03-29T10:35:31.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.053+0000 7f7394ff9640 1 -- 172.21.15.84:0/1918753838 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73980ac8d0 con 0x7f73981457f0 2024-03-29T10:35:31.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.053+0000 7f7394ff9640 1 -- 172.21.15.84:0/1918753838 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1218025221 0 0) 0x7f73980ac680 con 0x7f73980a9d70 2024-03-29T10:35:31.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.053+0000 7f7394ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:35:31.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.053+0000 7f7394ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:31.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.053+0000 7f7394ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:31.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.053+0000 7f7394ff9640 10 monclient(hunting): my global_id is 4252 2024-03-29T10:35:31.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.053+0000 7f7394ff9640 1 -- 172.21.15.84:0/1918753838 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7384004eb0 con 0x7f73980a9d70 2024-03-29T10:35:31.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.053+0000 7f7394ff9640 1 -- 172.21.15.84:0/1918753838 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 911871078 0 0) 0x7f7398137840 con 0x7f739813f650 2024-03-29T10:35:31.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.053+0000 7f7394ff9640 1 -- 172.21.15.84:0/1918753838 >> v1:172.21.15.84:6790/0 conn(0x7f73981457f0 legacy=0x7f73981f3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:31.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.053+0000 7f7394ff9640 1 -- 172.21.15.84:0/1918753838 >> v1:172.21.15.86:6789/0 conn(0x7f73980a9d70 legacy=0x7f73981448a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:31.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.053+0000 7f7394ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:35:31.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.053+0000 7f7394ff9640 1 monclient: found mon.a 2024-03-29T10:35:31.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.053+0000 7f7394ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:31.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.053+0000 7f7394ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:35:31.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.053+0000 7f7394ff9640 1 -- 172.21.15.84:0/1918753838 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f739808b9e0 con 0x7f739813f650 2024-03-29T10:35:31.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.053+0000 7f7394ff9640 10 monclient: _finish_auth 0 2024-03-29T10:35:31.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.053+0000 7f7394ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:35:31.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.053+0000 7f7394ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:31.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.053+0000 7f7394ff9640 1 -- 172.21.15.84:0/1918753838 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7390003970 con 0x7f739813f650 2024-03-29T10:35:31.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.053+0000 7f7394ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:31.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.053+0000 7f73967fc640 4 set_mon_vals no callback set 2024-03-29T10:35:31.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.054+0000 7f739e64e640 5 monclient: authenticate success, global_id 4254 2024-03-29T10:35:31.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.054+0000 7f739e64e640 10 monclient: _renew_subs 2024-03-29T10:35:31.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.054+0000 7f739e64e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:35:31.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.054+0000 7f739e64e640 1 -- 172.21.15.84:0/1918753838 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73980ab660 con 0x7f739813f650 2024-03-29T10:35:31.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.054+0000 7f739e64e640 10 client.4254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:35:31.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.054+0000 7f739e64e640 10 monclient: _renew_subs 2024-03-29T10:35:31.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.054+0000 7f739e64e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:35:31.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.054+0000 7f739e64e640 1 -- 172.21.15.84:0/1918753838 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f73980b15b0 con 0x7f739813f650 2024-03-29T10:35:31.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.054+0000 7f739e64e640 1 librados: init done 2024-03-29T10:35:31.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.055+0000 7f7394ff9640 1 -- 172.21.15.84:0/1918753838 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f7390003c80 con 0x7f739813f650 2024-03-29T10:35:31.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.055+0000 7f7394ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:31.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.055+0000 7f7394ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:35:31.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.055+0000 7f7394ff9640 10 monclient: dump: 2024-03-29T10:35:31.044 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:31.044 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:31.044 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:31.044 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:31.044 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:31.044 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:31.044 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:31.044 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:31.044 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:31.044 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:31.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.056+0000 7f739e64e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:31.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.056+0000 7f739e64e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:31.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.056+0000 7f739e64e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:35:31.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.056+0000 7f739e64e640 1 -- 172.21.15.84:0/1918753838 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7398075f10 con 0x7f739813f650 2024-03-29T10:35:31.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.058+0000 7f7394ff9640 1 -- 172.21.15.84:0/1918753838 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f7390019c50 con 0x7f739813f650 2024-03-29T10:35:31.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.058+0000 7f7394ff9640 1 -- 172.21.15.84:0/1918753838 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f7390083b10 con 0x7f739813f650 2024-03-29T10:35:31.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.058+0000 7f7394ff9640 10 client.4254.objecter ms_dispatch 0x7f7398000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:35:31.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.058+0000 7f7394ff9640 3 client.4254.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:35:31.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.058+0000 7f7394ff9640 3 client.4254.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:35:31.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.059+0000 7f7394ff9640 20 client.4254.objecter dump_active .. 0 homeless 2024-03-29T10:35:31.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.059+0000 7f7397fff640 10 monclient: get_auth_request con 0x7f7384076620 auth_method 0 2024-03-29T10:35:31.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.061+0000 7f7394ff9640 10 client.4254.objecter ms_handle_connect 0x7f7384076620 2024-03-29T10:35:31.054 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.068+0000 7f7394ff9640 1 -- 172.21.15.84:0/1918753838 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f739004f3f0 con 0x7f739813f650 2024-03-29T10:35:31.054 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.069+0000 7f7394ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:31.054 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.069+0000 7f7394ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:35:31.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.392+0000 7f739e64e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:31.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.392+0000 7f739e64e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:31.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.392+0000 7f739e64e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:35:31.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.392+0000 7f739e64e640 1 -- 172.21.15.84:0/1918753838 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f739808bb70 con 0x7f739813f650 2024-03-29T10:35:31.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.395+0000 7f7394ff9640 1 -- 172.21.15.84:0/1918753838 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f7390051320 con 0x7f739813f650 2024-03-29T10:35:31.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.395+0000 7f7394ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:31.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.395+0000 7f7394ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:35:31.381 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:35:31.381 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:35:31.384 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.399+0000 7f739e64e640 10 librados: watch_flush enter 2024-03-29T10:35:31.384 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.399+0000 7f739e64e640 10 librados: watch_flush exit 2024-03-29T10:35:31.384 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.399+0000 7f739e64e640 20 client.4254.objecter shutdown clearing up homeless session... 2024-03-29T10:35:31.384 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.399+0000 7f739e64e640 10 client.4254.objecter successfully canceled tick 2024-03-29T10:35:31.384 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.399+0000 7f739e64e640 1 -- 172.21.15.84:0/1918753838 >> v1:172.21.15.86:6816/3337225337 conn(0x7f7384076620 legacy=0x7f7384078ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:31.384 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.399+0000 7f739e64e640 10 monclient: shutdown 2024-03-29T10:35:31.384 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.399+0000 7f739e64e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:31.384 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.399+0000 7f739e64e640 1 -- 172.21.15.84:0/1918753838 >> v1:172.21.15.84:6789/0 conn(0x7f739813f650 legacy=0x7f7398141c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:31.385 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.400+0000 7f739e64e640 1 -- 172.21.15.84:0/1918753838 shutdown_connections 2024-03-29T10:35:31.385 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.400+0000 7f739e64e640 1 -- 172.21.15.84:0/1918753838 >> 172.21.15.84:0/1918753838 conn(0x7f7398066a00 msgr2=0x7f73980b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:31.385 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.400+0000 7f739e64e640 1 -- 172.21.15.84:0/1918753838 shutdown_connections 2024-03-29T10:35:31.385 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.400+0000 7f739e64e640 1 -- 172.21.15.84:0/1918753838 wait complete. 2024-03-29T10:35:31.385 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:31.400+0000 7f739e64e640 1 librados: shutdown 2024-03-29T10:35:31.403 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:35:38.211 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:35:38.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.374+0000 7fed4b4d9640 10 monclient: get_monmap_and_config 2024-03-29T10:35:38.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.374+0000 7fed4b4d9640 10 monclient: build_initial_monmap 2024-03-29T10:35:38.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.374+0000 7fed4b4d9640 10 monclient: monmap: 2024-03-29T10:35:38.359 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:35:38.359 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:38.359 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:35:38.375101+0000 2024-03-29T10:35:38.359 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:35:38.375101+0000 2024-03-29T10:35:38.359 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:38.359 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:38.359 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:35:38.360 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:35:38.360 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:35:38.360 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:38.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.375+0000 7fed4b4d9640 1 Processor -- start 2024-03-29T10:35:38.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.375+0000 7fed4b4d9640 1 -- start start 2024-03-29T10:35:38.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.375+0000 7fed4b4d9640 10 monclient: init 2024-03-29T10:35:38.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.376+0000 7fed4b4d9640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:38.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.376+0000 7fed4b4d9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:35:38.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.376+0000 7fed4b4d9640 10 monclient(hunting): picked mon.noname-c con 0x7fed4406ee40 addr v1:172.21.15.86:6789/0 2024-03-29T10:35:38.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.376+0000 7fed4b4d9640 10 monclient(hunting): picked mon.noname-b con 0x7fed440aeda0 addr v1:172.21.15.84:6790/0 2024-03-29T10:35:38.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.376+0000 7fed4b4d9640 10 monclient(hunting): picked mon.noname-a con 0x7fed4413c470 addr v1:172.21.15.84:6789/0 2024-03-29T10:35:38.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.376+0000 7fed4b4d9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed4406fd20 con 0x7fed4413c470 2024-03-29T10:35:38.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.376+0000 7fed4b4d9640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed44071090 con 0x7fed4406ee40 2024-03-29T10:35:38.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.376+0000 7fed4b4d9640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed44071370 con 0x7fed440aeda0 2024-03-29T10:35:38.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.376+0000 7fed4b4d9640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:38.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.376+0000 7fed4b4d9640 10 monclient(hunting): authenticate will time out at 840.639765s 2024-03-29T10:35:38.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.380+0000 7fed49275640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fed4406ee40 0x7fed4413bd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:35375/0 (socket says 172.21.15.84:35375) 2024-03-29T10:35:38.365 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.380+0000 7fed49275640 1 -- 172.21.15.84:0/1386186550 learned_addr learned my addr 172.21.15.84:0/1386186550 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:35:38.365 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.380+0000 7fed3bfff640 1 -- 172.21.15.84:0/1386186550 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 864460213 0 0) 0x7fed44071090 con 0x7fed4406ee40 2024-03-29T10:35:38.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.380+0000 7fed3bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:35:38.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.380+0000 7fed3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:38.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.380+0000 7fed3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:38.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.381+0000 7fed3bfff640 10 monclient(hunting): my global_id is 4255 2024-03-29T10:35:38.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.381+0000 7fed3bfff640 1 -- 172.21.15.84:0/1386186550 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed30003520 con 0x7fed4406ee40 2024-03-29T10:35:38.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.383+0000 7fed3bfff640 1 -- 172.21.15.84:0/1386186550 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1202220632 0 0) 0x7fed44071370 con 0x7fed440aeda0 2024-03-29T10:35:38.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.384+0000 7fed3bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:35:38.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.384+0000 7fed3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:38.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.384+0000 7fed3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:38.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.384+0000 7fed3bfff640 10 monclient(hunting): my global_id is 4262 2024-03-29T10:35:38.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.384+0000 7fed3bfff640 1 -- 172.21.15.84:0/1386186550 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed44071090 con 0x7fed440aeda0 2024-03-29T10:35:38.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.384+0000 7fed3bfff640 1 -- 172.21.15.84:0/1386186550 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3198326831 0 0) 0x7fed4406fd20 con 0x7fed4413c470 2024-03-29T10:35:38.374 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.384+0000 7fed3bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:35:38.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.384+0000 7fed3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:38.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.384+0000 7fed3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:38.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.384+0000 7fed3bfff640 10 monclient(hunting): my global_id is 4257 2024-03-29T10:35:38.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.384+0000 7fed3bfff640 1 -- 172.21.15.84:0/1386186550 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed44071370 con 0x7fed4413c470 2024-03-29T10:35:38.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.384+0000 7fed3bfff640 1 -- 172.21.15.84:0/1386186550 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3573871351 0 0) 0x7fed30003520 con 0x7fed4406ee40 2024-03-29T10:35:38.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.384+0000 7fed3bfff640 1 -- 172.21.15.84:0/1386186550 >> v1:172.21.15.84:6790/0 conn(0x7fed440aeda0 legacy=0x7fed44071790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:38.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.384+0000 7fed3bfff640 1 -- 172.21.15.84:0/1386186550 >> v1:172.21.15.84:6789/0 conn(0x7fed4413c470 legacy=0x7fed4414b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:38.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.384+0000 7fed3bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:35:38.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.384+0000 7fed3bfff640 1 monclient: found mon.noname-c 2024-03-29T10:35:38.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.385+0000 7fed3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:38.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.385+0000 7fed3bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-29T10:35:38.375 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.385+0000 7fed3bfff640 1 -- 172.21.15.84:0/1386186550 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed44070220 con 0x7fed4406ee40 2024-03-29T10:35:38.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.385+0000 7fed3bfff640 10 monclient: _finish_auth 0 2024-03-29T10:35:38.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.385+0000 7fed3bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:35:38.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.385+0000 7fed3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:38.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.385+0000 7fed4b4d9640 5 monclient: authenticate success, global_id 4255 2024-03-29T10:35:38.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.385+0000 7fed4b4d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:35:38.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.385+0000 7fed3bfff640 1 -- 172.21.15.84:0/1386186550 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fed34001690 con 0x7fed4406ee40 2024-03-29T10:35:38.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.385+0000 7fed3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:38.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.385+0000 7fed3bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:35:38.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.385+0000 7fed3bfff640 10 monclient: dump: 2024-03-29T10:35:38.376 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:38.377 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:38.377 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:38.377 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:38.377 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:38.377 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:38.377 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:38.377 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:38.377 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:38.377 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:38.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.385+0000 7fed4b4d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:35:38.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.385+0000 7fed3bfff640 1 -- 172.21.15.84:0/1386186550 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed340018b0 con 0x7fed4406ee40 2024-03-29T10:35:38.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.385+0000 7fed3bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:38.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.385+0000 7fed4b4d9640 10 monclient: get_monmap_and_config success 2024-03-29T10:35:38.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.385+0000 7fed4b4d9640 4 set_mon_vals no callback set 2024-03-29T10:35:38.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.386+0000 7fed3bfff640 1 -- 172.21.15.84:0/1386186550 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fed34001c40 con 0x7fed4406ee40 2024-03-29T10:35:38.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.386+0000 7fed3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:38.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.386+0000 7fed3bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:35:38.378 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.386+0000 7fed3bfff640 10 monclient: dump: 2024-03-29T10:35:38.378 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:38.378 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:38.378 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:38.378 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:38.379 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:38.379 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:38.379 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:38.379 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:38.379 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:38.379 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:38.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.386+0000 7fed4b4d9640 10 monclient: shutdown 2024-03-29T10:35:38.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.386+0000 7fed4b4d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:38.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.386+0000 7fed4b4d9640 1 -- 172.21.15.84:0/1386186550 >> v1:172.21.15.86:6789/0 conn(0x7fed4406ee40 legacy=0x7fed4413bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:38.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.386+0000 7fed4b4d9640 1 -- 172.21.15.84:0/1386186550 shutdown_connections 2024-03-29T10:35:38.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.386+0000 7fed4b4d9640 1 -- 172.21.15.84:0/1386186550 >> 172.21.15.84:0/1386186550 conn(0x7fed44067470 msgr2=0x7fed440b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:38.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.386+0000 7fed4b4d9640 1 -- 172.21.15.84:0/1386186550 shutdown_connections 2024-03-29T10:35:38.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.387+0000 7fed4b4d9640 1 -- 172.21.15.84:0/1386186550 wait complete. 2024-03-29T10:35:38.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.387+0000 7fed4b4d9640 10 monclient: build_initial_monmap 2024-03-29T10:35:38.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.387+0000 7fed4b4d9640 10 monclient: monmap: 2024-03-29T10:35:38.380 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:35:38.380 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:35:38.380 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:35:38.380 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:35:38.380 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:38.380 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:38.380 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:35:38.380 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:35:38.381 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:35:38.381 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:38.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.388+0000 7fed4b4d9640 1 librados: starting msgr at 2024-03-29T10:35:38.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.388+0000 7fed4b4d9640 1 librados: starting objecter 2024-03-29T10:35:38.381 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.388+0000 7fed4b4d9640 1 Processor -- start 2024-03-29T10:35:38.386 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.388+0000 7fed4b4d9640 1 -- start start 2024-03-29T10:35:38.386 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.388+0000 7fed4b4d9640 1 librados: setting wanted keys 2024-03-29T10:35:38.386 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.388+0000 7fed4b4d9640 1 librados: calling monclient init 2024-03-29T10:35:38.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.388+0000 7fed4b4d9640 10 monclient: init 2024-03-29T10:35:38.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.389+0000 7fed4b4d9640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:38.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.389+0000 7fed4b4d9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:35:38.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.389+0000 7fed4b4d9640 10 monclient(hunting): picked mon.noname-a con 0x7fed4406ee40 addr v1:172.21.15.84:6789/0 2024-03-29T10:35:38.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.389+0000 7fed4b4d9640 10 monclient(hunting): picked mon.noname-c con 0x7fed440aeda0 addr v1:172.21.15.84:6790/0 2024-03-29T10:35:38.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.389+0000 7fed4b4d9640 10 monclient(hunting): picked mon.noname-b con 0x7fed4413c470 addr v1:172.21.15.86:6789/0 2024-03-29T10:35:38.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.389+0000 7fed4b4d9640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed4412ff30 con 0x7fed4406ee40 2024-03-29T10:35:38.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.389+0000 7fed4b4d9640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed44140c10 con 0x7fed4413c470 2024-03-29T10:35:38.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.389+0000 7fed4b4d9640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed44140ef0 con 0x7fed440aeda0 2024-03-29T10:35:38.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.389+0000 7fed4b4d9640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:38.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.389+0000 7fed4b4d9640 10 monclient(hunting): authenticate will time out at 840.652610s 2024-03-29T10:35:38.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.393+0000 7fed49a76640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fed4413c470 0x7fed4413d310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:35445/0 (socket says 172.21.15.84:35445) 2024-03-29T10:35:38.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.393+0000 7fed49a76640 1 -- 172.21.15.84:0/3008821523 learned_addr learned my addr 172.21.15.84:0/3008821523 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:35:38.387 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.395+0000 7fed39ffb640 10 client.?.objecter ms_handle_connect 0x7fed4413c470 2024-03-29T10:35:38.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.395+0000 7fed39ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:38.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.396+0000 7fed39ffb640 1 -- 172.21.15.84:0/3008821523 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1409634466 0 0) 0x7fed44140c10 con 0x7fed4413c470 2024-03-29T10:35:38.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.396+0000 7fed39ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:35:38.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.396+0000 7fed39ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:38.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.396+0000 7fed39ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:38.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.396+0000 7fed39ffb640 10 monclient(hunting): my global_id is 4258 2024-03-29T10:35:38.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.396+0000 7fed39ffb640 1 -- 172.21.15.84:0/3008821523 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed28004eb0 con 0x7fed4413c470 2024-03-29T10:35:38.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.396+0000 7fed39ffb640 10 client.?.objecter ms_handle_connect 0x7fed440aeda0 2024-03-29T10:35:38.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.396+0000 7fed39ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:38.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.396+0000 7fed39ffb640 1 -- 172.21.15.84:0/3008821523 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1870965512 0 0) 0x7fed28004eb0 con 0x7fed4413c470 2024-03-29T10:35:38.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.397+0000 7fed39ffb640 1 -- 172.21.15.84:0/3008821523 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed44140c10 con 0x7fed4413c470 2024-03-29T10:35:38.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.397+0000 7fed39ffb640 10 client.?.objecter ms_handle_connect 0x7fed4406ee40 2024-03-29T10:35:38.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.397+0000 7fed39ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:38.388 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.397+0000 7fed39ffb640 1 -- 172.21.15.84:0/3008821523 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fed400018b0 con 0x7fed4413c470 2024-03-29T10:35:38.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.397+0000 7fed39ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:38.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.397+0000 7fed39ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:35:38.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.397+0000 7fed39ffb640 10 monclient(hunting): dump: 2024-03-29T10:35:38.389 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:38.389 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:38.389 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:38.389 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:38.389 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:38.389 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:38.389 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:38.389 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:38.389 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:38.389 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:38.389 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.397+0000 7fed39ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:35:38.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.397+0000 7fed39ffb640 1 -- 172.21.15.84:0/3008821523 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 957092359 0 0) 0x7fed44140ef0 con 0x7fed440aeda0 2024-03-29T10:35:38.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.397+0000 7fed39ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:35:38.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.397+0000 7fed39ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:38.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.397+0000 7fed39ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:38.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.397+0000 7fed39ffb640 10 monclient(hunting): my global_id is 4265 2024-03-29T10:35:38.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.397+0000 7fed39ffb640 1 -- 172.21.15.84:0/3008821523 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed28004eb0 con 0x7fed440aeda0 2024-03-29T10:35:38.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.398+0000 7fed39ffb640 1 -- 172.21.15.84:0/3008821523 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 907542582 0 0) 0x7fed44140c10 con 0x7fed4413c470 2024-03-29T10:35:38.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.398+0000 7fed39ffb640 1 -- 172.21.15.84:0/3008821523 >> v1:172.21.15.84:6790/0 conn(0x7fed440aeda0 legacy=0x7fed4413cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:38.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.398+0000 7fed39ffb640 1 -- 172.21.15.84:0/3008821523 >> v1:172.21.15.84:6789/0 conn(0x7fed4406ee40 legacy=0x7fed440ada60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:38.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.398+0000 7fed39ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:35:38.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.398+0000 7fed39ffb640 1 monclient: found mon.b 2024-03-29T10:35:38.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.398+0000 7fed39ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:38.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.398+0000 7fed39ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:35:38.390 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.398+0000 7fed39ffb640 1 -- 172.21.15.84:0/3008821523 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed4412f1e0 con 0x7fed4413c470 2024-03-29T10:35:38.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.398+0000 7fed39ffb640 10 monclient: _finish_auth 0 2024-03-29T10:35:38.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.398+0000 7fed39ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:35:38.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.398+0000 7fed39ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:38.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.398+0000 7fed4b4d9640 5 monclient: authenticate success, global_id 4258 2024-03-29T10:35:38.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.398+0000 7fed4b4d9640 10 monclient: _renew_subs 2024-03-29T10:35:38.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.398+0000 7fed4b4d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:35:38.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.398+0000 7fed4b4d9640 1 -- 172.21.15.84:0/3008821523 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed4412f490 con 0x7fed4413c470 2024-03-29T10:35:38.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.399+0000 7fed4b4d9640 10 client.4258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:35:38.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.399+0000 7fed4b4d9640 10 monclient: _renew_subs 2024-03-29T10:35:38.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.399+0000 7fed4b4d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:35:38.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.399+0000 7fed4b4d9640 1 -- 172.21.15.84:0/3008821523 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed44142630 con 0x7fed4413c470 2024-03-29T10:35:38.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.399+0000 7fed4b4d9640 1 librados: init done 2024-03-29T10:35:38.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.399+0000 7fed39ffb640 1 -- 172.21.15.84:0/3008821523 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed40001a80 con 0x7fed4413c470 2024-03-29T10:35:38.391 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.399+0000 7fed39ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:38.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.399+0000 7fed39ffb640 1 -- 172.21.15.84:0/3008821523 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fed400038c0 con 0x7fed4413c470 2024-03-29T10:35:38.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.399+0000 7fed39ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:38.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.399+0000 7fed3b7fe640 4 set_mon_vals no callback set 2024-03-29T10:35:38.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.399+0000 7fed39ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:35:38.392 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.400+0000 7fed39ffb640 10 monclient: dump: 2024-03-29T10:35:38.392 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:38.392 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:38.392 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:38.392 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:38.392 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:38.392 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:38.392 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:38.392 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:38.392 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:38.392 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:38.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.402+0000 7fed39ffb640 1 -- 172.21.15.84:0/3008821523 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7fed40003a20 con 0x7fed4413c470 2024-03-29T10:35:38.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.402+0000 7fed39ffb640 1 -- 172.21.15.84:0/3008821523 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7fed40084a00 con 0x7fed4413c470 2024-03-29T10:35:38.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.402+0000 7fed39ffb640 10 client.4258.objecter ms_dispatch 0x7fed44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:35:38.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.402+0000 7fed39ffb640 3 client.4258.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:35:38.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.402+0000 7fed39ffb640 3 client.4258.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:35:38.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.402+0000 7fed39ffb640 20 client.4258.objecter dump_active .. 0 homeless 2024-03-29T10:35:38.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.403+0000 7fed4b4d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:38.393 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.403+0000 7fed4b4d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:38.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.403+0000 7fed4b4d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:35:38.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.403+0000 7fed4b4d9640 1 -- 172.21.15.84:0/3008821523 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed24005080 con 0x7fed4413c470 2024-03-29T10:35:38.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.403+0000 7fed49275640 10 monclient: get_auth_request con 0x7fed2806dd30 auth_method 0 2024-03-29T10:35:38.394 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.403+0000 7fed39ffb640 10 client.4258.objecter ms_handle_connect 0x7fed2806dd30 2024-03-29T10:35:38.397 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.412+0000 7fed39ffb640 1 -- 172.21.15.84:0/3008821523 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fed400502e0 con 0x7fed4413c470 2024-03-29T10:35:38.397 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.412+0000 7fed39ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:38.397 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.412+0000 7fed39ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:35:38.711 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.726+0000 7fed4b4d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:38.712 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.726+0000 7fed4b4d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:38.712 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.726+0000 7fed4b4d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:35:38.712 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.726+0000 7fed4b4d9640 1 -- 172.21.15.84:0/3008821523 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed24005330 con 0x7fed4413c470 2024-03-29T10:35:38.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.730+0000 7fed39ffb640 1 -- 172.21.15.84:0/3008821523 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7fed40052210 con 0x7fed4413c470 2024-03-29T10:35:38.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.730+0000 7fed39ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:38.715 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.730+0000 7fed39ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:35:38.715 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:35:38.715 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:35:38.718 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.733+0000 7fed4b4d9640 10 librados: watch_flush enter 2024-03-29T10:35:38.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.734+0000 7fed4b4d9640 10 librados: watch_flush exit 2024-03-29T10:35:38.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.734+0000 7fed4b4d9640 20 client.4258.objecter shutdown clearing up homeless session... 2024-03-29T10:35:38.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.734+0000 7fed4b4d9640 10 client.4258.objecter successfully canceled tick 2024-03-29T10:35:38.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.734+0000 7fed4b4d9640 1 -- 172.21.15.84:0/3008821523 >> v1:172.21.15.86:6816/3337225337 conn(0x7fed2806dd30 legacy=0x7fed280701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:38.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.734+0000 7fed4b4d9640 10 monclient: shutdown 2024-03-29T10:35:38.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.734+0000 7fed4b4d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:38.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.734+0000 7fed4b4d9640 1 -- 172.21.15.84:0/3008821523 >> v1:172.21.15.86:6789/0 conn(0x7fed4413c470 legacy=0x7fed4413d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:38.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.734+0000 7fed4b4d9640 1 -- 172.21.15.84:0/3008821523 shutdown_connections 2024-03-29T10:35:38.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.735+0000 7fed4b4d9640 1 -- 172.21.15.84:0/3008821523 >> 172.21.15.84:0/3008821523 conn(0x7fed44067470 msgr2=0x7fed440afe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:38.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.735+0000 7fed4b4d9640 1 -- 172.21.15.84:0/3008821523 shutdown_connections 2024-03-29T10:35:38.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.735+0000 7fed4b4d9640 1 -- 172.21.15.84:0/3008821523 wait complete. 2024-03-29T10:35:38.721 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:38.735+0000 7fed4b4d9640 1 librados: shutdown 2024-03-29T10:35:38.736 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:35:45.543 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:35:45.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.706+0000 7f290224e640 10 monclient: get_monmap_and_config 2024-03-29T10:35:45.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.706+0000 7f290224e640 10 monclient: build_initial_monmap 2024-03-29T10:35:45.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.706+0000 7f290224e640 10 monclient: monmap: 2024-03-29T10:35:45.692 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:35:45.692 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:45.692 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:35:45.707829+0000 2024-03-29T10:35:45.692 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:35:45.707829+0000 2024-03-29T10:35:45.692 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:45.692 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:45.692 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:35:45.692 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:35:45.692 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:35:45.692 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:45.694 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.709+0000 7f290224e640 1 Processor -- start 2024-03-29T10:35:45.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.710+0000 7f290224e640 1 -- start start 2024-03-29T10:35:45.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.710+0000 7f290224e640 10 monclient: init 2024-03-29T10:35:45.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.711+0000 7f290224e640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:45.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.711+0000 7f290224e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:35:45.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.711+0000 7f290224e640 10 monclient(hunting): picked mon.noname-c con 0x7f28fc1422b0 addr v1:172.21.15.86:6789/0 2024-03-29T10:35:45.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.711+0000 7f290224e640 10 monclient(hunting): picked mon.noname-b con 0x7f28fc142690 addr v1:172.21.15.84:6790/0 2024-03-29T10:35:45.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.711+0000 7f290224e640 10 monclient(hunting): picked mon.noname-a con 0x7f28fc0b17d0 addr v1:172.21.15.84:6789/0 2024-03-29T10:35:45.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.711+0000 7f290224e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28fc142c80 con 0x7f28fc0b17d0 2024-03-29T10:35:45.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.711+0000 7f290224e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28fc0aae40 con 0x7f28fc1422b0 2024-03-29T10:35:45.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.711+0000 7f290224e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28fc13fbd0 con 0x7f28fc142690 2024-03-29T10:35:45.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.711+0000 7f290224e640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:45.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.711+0000 7f290224e640 10 monclient(hunting): authenticate will time out at 847.974897s 2024-03-29T10:35:45.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.715+0000 7f28fbfff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f28fc0b17d0 0x7f28fc0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:46033/0 (socket says 172.21.15.84:46033) 2024-03-29T10:35:45.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.715+0000 7f28fbfff640 1 -- 172.21.15.84:0/1879462337 learned_addr learned my addr 172.21.15.84:0/1879462337 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:35:45.701 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.716+0000 7f28fa7fc640 1 -- 172.21.15.84:0/1879462337 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3844405768 0 0) 0x7f28fc142c80 con 0x7f28fc0b17d0 2024-03-29T10:35:45.701 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.716+0000 7f28fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:35:45.701 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.716+0000 7f28fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:45.701 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.716+0000 7f28fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:45.701 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.716+0000 7f28fa7fc640 10 monclient(hunting): my global_id is 4263 2024-03-29T10:35:45.701 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.716+0000 7f28fa7fc640 1 -- 172.21.15.84:0/1879462337 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28e4003520 con 0x7f28fc0b17d0 2024-03-29T10:35:45.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.717+0000 7f28fa7fc640 1 -- 172.21.15.84:0/1879462337 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2488107883 0 0) 0x7f28e4003520 con 0x7f28fc0b17d0 2024-03-29T10:35:45.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.717+0000 7f28fa7fc640 1 -- 172.21.15.84:0/1879462337 >> v1:172.21.15.84:6790/0 conn(0x7f28fc142690 legacy=0x7f28fc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:35:45.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.717+0000 7f28fa7fc640 1 -- 172.21.15.84:0/1879462337 >> v1:172.21.15.86:6789/0 conn(0x7f28fc1422b0 legacy=0x7f28fc13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:35:45.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.718+0000 7f28fa7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:35:45.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.718+0000 7f28fa7fc640 1 monclient: found mon.noname-a 2024-03-29T10:35:45.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.718+0000 7f28fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:45.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.718+0000 7f28fa7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T10:35:45.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.718+0000 7f28fa7fc640 1 -- 172.21.15.84:0/1879462337 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28fc151940 con 0x7f28fc0b17d0 2024-03-29T10:35:45.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.718+0000 7f28fa7fc640 10 monclient: _finish_auth 0 2024-03-29T10:35:45.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.718+0000 7f28fa7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:35:45.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.718+0000 7f28fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:45.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.718+0000 7f28fa7fc640 1 -- 172.21.15.84:0/1879462337 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f28f0001690 con 0x7f28fc0b17d0 2024-03-29T10:35:45.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.718+0000 7f28fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:45.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.718+0000 7f28fa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:35:45.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.718+0000 7f28fa7fc640 10 monclient: dump: 2024-03-29T10:35:45.705 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:45.705 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:45.705 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:45.705 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:45.705 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:45.705 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:45.705 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:45.705 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:45.705 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:45.705 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:45.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.718+0000 7f290224e640 5 monclient: authenticate success, global_id 4263 2024-03-29T10:35:45.706 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.718+0000 7f290224e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:35:45.706 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.720+0000 7f28fa7fc640 1 -- 172.21.15.84:0/1879462337 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28f00017f0 con 0x7f28fc0b17d0 2024-03-29T10:35:45.706 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.720+0000 7f28fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:45.706 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.720+0000 7f28fa7fc640 1 -- 172.21.15.84:0/1879462337 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f28f0001c40 con 0x7f28fc0b17d0 2024-03-29T10:35:45.706 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.720+0000 7f28fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:45.706 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.720+0000 7f28fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:35:45.707 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.720+0000 7f28fa7fc640 10 monclient: dump: 2024-03-29T10:35:45.707 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:45.707 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:45.707 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:45.707 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:45.707 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:45.707 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:45.707 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:45.707 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:45.707 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:45.707 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:45.707 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.720+0000 7f290224e640 10 monclient: get_monmap_and_config success 2024-03-29T10:35:45.707 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.720+0000 7f290224e640 4 set_mon_vals no callback set 2024-03-29T10:35:45.707 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.721+0000 7f290224e640 10 monclient: shutdown 2024-03-29T10:35:45.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.721+0000 7f290224e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:45.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.721+0000 7f290224e640 1 -- 172.21.15.84:0/1879462337 >> v1:172.21.15.84:6789/0 conn(0x7f28fc0b17d0 legacy=0x7f28fc0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:45.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.721+0000 7f290224e640 1 -- 172.21.15.84:0/1879462337 shutdown_connections 2024-03-29T10:35:45.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.721+0000 7f290224e640 1 -- 172.21.15.84:0/1879462337 >> 172.21.15.84:0/1879462337 conn(0x7f28fc066a00 msgr2=0x7f28fc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:45.709 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.721+0000 7f290224e640 1 -- 172.21.15.84:0/1879462337 shutdown_connections 2024-03-29T10:35:45.710 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.721+0000 7f290224e640 1 -- 172.21.15.84:0/1879462337 wait complete. 2024-03-29T10:35:45.710 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.722+0000 7f290224e640 10 monclient: build_initial_monmap 2024-03-29T10:35:45.710 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.722+0000 7f290224e640 10 monclient: monmap: 2024-03-29T10:35:45.710 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:35:45.710 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:35:45.710 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:35:45.710 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:35:45.710 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:45.710 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:45.710 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:35:45.710 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:35:45.711 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:35:45.711 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:45.711 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.722+0000 7f290224e640 1 librados: starting msgr at 2024-03-29T10:35:45.711 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.722+0000 7f290224e640 1 librados: starting objecter 2024-03-29T10:35:45.711 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.722+0000 7f290224e640 1 Processor -- start 2024-03-29T10:35:45.711 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.723+0000 7f290224e640 1 -- start start 2024-03-29T10:35:45.711 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.723+0000 7f290224e640 1 librados: setting wanted keys 2024-03-29T10:35:45.711 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.723+0000 7f290224e640 1 librados: calling monclient init 2024-03-29T10:35:45.711 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.723+0000 7f290224e640 10 monclient: init 2024-03-29T10:35:45.711 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.723+0000 7f290224e640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:45.711 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.723+0000 7f290224e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:35:45.711 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.723+0000 7f290224e640 10 monclient(hunting): picked mon.noname-b con 0x7f28fc0b17d0 addr v1:172.21.15.86:6789/0 2024-03-29T10:35:45.711 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.723+0000 7f290224e640 10 monclient(hunting): picked mon.noname-a con 0x7f28fc1422b0 addr v1:172.21.15.84:6789/0 2024-03-29T10:35:45.711 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.723+0000 7f290224e640 10 monclient(hunting): picked mon.noname-c con 0x7f28fc142690 addr v1:172.21.15.84:6790/0 2024-03-29T10:35:45.712 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.723+0000 7f290224e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28fc0ab670 con 0x7f28fc1422b0 2024-03-29T10:35:45.712 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.723+0000 7f290224e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28fc0ac4a0 con 0x7f28fc0b17d0 2024-03-29T10:35:45.712 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.723+0000 7f290224e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28fc0ac780 con 0x7f28fc142690 2024-03-29T10:35:45.712 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.723+0000 7f290224e640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:45.712 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.723+0000 7f290224e640 10 monclient(hunting): authenticate will time out at 847.987020s 2024-03-29T10:35:45.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.728+0000 7f28faffd640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f28fc1422b0 0x7f28fc132530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:46133/0 (socket says 172.21.15.84:46133) 2024-03-29T10:35:45.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.728+0000 7f28faffd640 1 -- 172.21.15.84:0/265945691 learned_addr learned my addr 172.21.15.84:0/265945691 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:35:45.716 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.729+0000 7f28e3fff640 10 client.?.objecter ms_handle_connect 0x7f28fc1422b0 2024-03-29T10:35:45.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.729+0000 7f28e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:45.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.730+0000 7f28e3fff640 1 -- 172.21.15.84:0/265945691 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613710966 0 0) 0x7f28fc0ab670 con 0x7f28fc1422b0 2024-03-29T10:35:45.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.730+0000 7f28e3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:35:45.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.730+0000 7f28e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:45.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.730+0000 7f28e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:45.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.730+0000 7f28e3fff640 10 monclient(hunting): my global_id is 4266 2024-03-29T10:35:45.717 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.730+0000 7f28e3fff640 1 -- 172.21.15.84:0/265945691 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28e8004eb0 con 0x7f28fc1422b0 2024-03-29T10:35:45.718 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.731+0000 7f28e3fff640 10 client.?.objecter ms_handle_connect 0x7f28fc0b17d0 2024-03-29T10:35:45.718 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.731+0000 7f28e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:45.718 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.731+0000 7f28e3fff640 1 -- 172.21.15.84:0/265945691 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 520601952 0 0) 0x7f28e8004eb0 con 0x7f28fc1422b0 2024-03-29T10:35:45.719 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.731+0000 7f28e3fff640 1 -- 172.21.15.84:0/265945691 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f28fc0ab670 con 0x7f28fc1422b0 2024-03-29T10:35:45.719 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.731+0000 7f28e3fff640 1 -- 172.21.15.84:0/265945691 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f28ec0017d0 con 0x7f28fc1422b0 2024-03-29T10:35:45.719 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.731+0000 7f28e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:45.719 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.731+0000 7f28e3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:35:45.719 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.731+0000 7f28e3fff640 10 monclient(hunting): dump: 2024-03-29T10:35:45.719 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:45.719 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:45.720 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:45.720 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:45.720 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:45.720 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:45.720 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:45.720 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:45.720 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:45.720 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:45.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.731+0000 7f28e3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:35:45.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.732+0000 7f28e3fff640 1 -- 172.21.15.84:0/265945691 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 931536687 0 0) 0x7f28fc0ac4a0 con 0x7f28fc0b17d0 2024-03-29T10:35:45.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.732+0000 7f28e3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:35:45.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.732+0000 7f28e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:45.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.732+0000 7f28e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:45.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.732+0000 7f28e3fff640 10 monclient(hunting): my global_id is 4261 2024-03-29T10:35:45.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.732+0000 7f28e3fff640 1 -- 172.21.15.84:0/265945691 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28e8004eb0 con 0x7f28fc0b17d0 2024-03-29T10:35:45.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.732+0000 7f28e3fff640 10 client.?.objecter ms_handle_connect 0x7f28fc142690 2024-03-29T10:35:45.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.732+0000 7f28e3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:45.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.732+0000 7f28e3fff640 1 -- 172.21.15.84:0/265945691 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1579289438 0 0) 0x7f28fc0ab670 con 0x7f28fc1422b0 2024-03-29T10:35:45.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.732+0000 7f28e3fff640 1 -- 172.21.15.84:0/265945691 >> v1:172.21.15.84:6790/0 conn(0x7f28fc142690 legacy=0x7f28fc08b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:45.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.732+0000 7f28e3fff640 1 -- 172.21.15.84:0/265945691 >> v1:172.21.15.86:6789/0 conn(0x7f28fc0b17d0 legacy=0x7f28fc12df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:45.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.732+0000 7f28e3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:35:45.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.732+0000 7f28e3fff640 1 monclient: found mon.a 2024-03-29T10:35:45.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.732+0000 7f28e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:45.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.732+0000 7f28e3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:35:45.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.732+0000 7f28e3fff640 1 -- 172.21.15.84:0/265945691 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28fc0aca60 con 0x7f28fc1422b0 2024-03-29T10:35:45.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.732+0000 7f28e3fff640 10 monclient: _finish_auth 0 2024-03-29T10:35:45.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.732+0000 7f28e3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:35:45.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.732+0000 7f28e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:45.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.732+0000 7f290224e640 5 monclient: authenticate success, global_id 4266 2024-03-29T10:35:45.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.732+0000 7f290224e640 10 monclient: _renew_subs 2024-03-29T10:35:45.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.732+0000 7f290224e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:35:45.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.732+0000 7f290224e640 1 -- 172.21.15.84:0/265945691 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28fc08bc20 con 0x7f28fc1422b0 2024-03-29T10:35:45.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.732+0000 7f290224e640 10 client.4266.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:35:45.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.732+0000 7f290224e640 10 monclient: _renew_subs 2024-03-29T10:35:45.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.732+0000 7f290224e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:35:45.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.732+0000 7f290224e640 1 -- 172.21.15.84:0/265945691 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f28fc131d00 con 0x7f28fc1422b0 2024-03-29T10:35:45.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.732+0000 7f290224e640 1 librados: init done 2024-03-29T10:35:45.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.733+0000 7f28e3fff640 1 -- 172.21.15.84:0/265945691 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28ec001d60 con 0x7f28fc1422b0 2024-03-29T10:35:45.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.733+0000 7f28e3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:45.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.733+0000 7f28e3fff640 1 -- 172.21.15.84:0/265945691 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f28ec004a00 con 0x7f28fc1422b0 2024-03-29T10:35:45.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.733+0000 7f28e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:45.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.733+0000 7f28e3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:35:45.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.733+0000 7f28e3fff640 10 monclient: dump: 2024-03-29T10:35:45.726 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:45.726 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:45.727 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:45.727 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:45.727 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:45.727 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:45.727 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:45.727 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:45.727 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:45.727 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:45.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.733+0000 7f28f9ffb640 4 set_mon_vals no callback set 2024-03-29T10:35:45.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.734+0000 7f28e1ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:45.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.736+0000 7f28e1ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:45.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.736+0000 7f28e1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:35:45.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.736+0000 7f28e1ffb640 1 -- 172.21.15.84:0/265945691 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28fc075f10 con 0x7f28fc1422b0 2024-03-29T10:35:45.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.736+0000 7f28e3fff640 1 -- 172.21.15.84:0/265945691 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f28ec01aa90 con 0x7f28fc1422b0 2024-03-29T10:35:45.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.738+0000 7f28e3fff640 1 -- 172.21.15.84:0/265945691 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f28ec085ae0 con 0x7f28fc1422b0 2024-03-29T10:35:45.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.738+0000 7f28e3fff640 10 client.4266.objecter ms_dispatch 0x7f28fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:35:45.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.738+0000 7f28e3fff640 3 client.4266.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:35:45.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.739+0000 7f28e3fff640 3 client.4266.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:35:45.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.739+0000 7f28e3fff640 20 client.4266.objecter dump_active .. 0 homeless 2024-03-29T10:35:45.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.739+0000 7f28fb7fe640 10 monclient: get_auth_request con 0x7f28e806dd40 auth_method 0 2024-03-29T10:35:45.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.743+0000 7f28e3fff640 10 client.4266.objecter ms_handle_connect 0x7f28e806dd40 2024-03-29T10:35:45.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.746+0000 7f28e3fff640 1 -- 172.21.15.84:0/265945691 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f28ec050390 con 0x7f28fc1422b0 2024-03-29T10:35:45.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.746+0000 7f28e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:45.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:45.746+0000 7f28e3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:35:46.063 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:46.078+0000 7f28e1ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:46.063 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:46.078+0000 7f28e1ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:46.063 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:46.078+0000 7f28e1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:35:46.063 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:46.078+0000 7f28e1ffb640 1 -- 172.21.15.84:0/265945691 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28fc0ab820 con 0x7f28fc1422b0 2024-03-29T10:35:46.064 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:46.079+0000 7f28e3fff640 1 -- 172.21.15.84:0/265945691 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f28fc0ab820 con 0x7f28fc1422b0 2024-03-29T10:35:46.064 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:46.079+0000 7f28e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:46.064 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:46.079+0000 7f28e3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:35:46.064 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:35:46.065 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:35:46.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:46.083+0000 7f28e1ffb640 10 librados: watch_flush enter 2024-03-29T10:35:46.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:46.084+0000 7f28e1ffb640 10 librados: watch_flush exit 2024-03-29T10:35:46.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:46.084+0000 7f28e1ffb640 20 client.4266.objecter shutdown clearing up homeless session... 2024-03-29T10:35:46.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:46.084+0000 7f28e1ffb640 10 client.4266.objecter successfully canceled tick 2024-03-29T10:35:46.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:46.084+0000 7f28e1ffb640 1 -- 172.21.15.84:0/265945691 >> v1:172.21.15.86:6816/3337225337 conn(0x7f28e806dd40 legacy=0x7f28e8070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:46.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:46.084+0000 7f28e1ffb640 10 monclient: shutdown 2024-03-29T10:35:46.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:46.084+0000 7f28e1ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:46.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:46.084+0000 7f28e1ffb640 1 -- 172.21.15.84:0/265945691 >> v1:172.21.15.84:6789/0 conn(0x7f28fc1422b0 legacy=0x7f28fc132530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:46.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:46.084+0000 7f28e1ffb640 1 -- 172.21.15.84:0/265945691 shutdown_connections 2024-03-29T10:35:46.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:46.084+0000 7f28e1ffb640 1 -- 172.21.15.84:0/265945691 >> 172.21.15.84:0/265945691 conn(0x7f28fc066a00 msgr2=0x7f28fc13d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:46.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:46.086+0000 7f28e1ffb640 1 -- 172.21.15.84:0/265945691 shutdown_connections 2024-03-29T10:35:46.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:46.087+0000 7f28e1ffb640 1 -- 172.21.15.84:0/265945691 wait complete. 2024-03-29T10:35:46.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:46.088+0000 7f28e1ffb640 1 librados: shutdown 2024-03-29T10:35:46.095 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:35:52.903 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:35:53.050 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.065+0000 7f727212b640 10 monclient: get_monmap_and_config 2024-03-29T10:35:53.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.065+0000 7f727212b640 10 monclient: build_initial_monmap 2024-03-29T10:35:53.051 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.065+0000 7f727212b640 10 monclient: monmap: 2024-03-29T10:35:53.051 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:35:53.051 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:53.051 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:35:53.066706+0000 2024-03-29T10:35:53.051 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:35:53.066706+0000 2024-03-29T10:35:53.051 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:53.051 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:53.051 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:35:53.051 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:35:53.052 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:35:53.052 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:53.052 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.067+0000 7f727212b640 1 Processor -- start 2024-03-29T10:35:53.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.067+0000 7f727212b640 1 -- start start 2024-03-29T10:35:53.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.067+0000 7f727212b640 10 monclient: init 2024-03-29T10:35:53.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.068+0000 7f727212b640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:53.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.068+0000 7f727212b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:35:53.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.068+0000 7f727212b640 10 monclient(hunting): picked mon.noname-c con 0x7f726c0708a0 addr v1:172.21.15.86:6789/0 2024-03-29T10:35:53.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.068+0000 7f727212b640 10 monclient(hunting): picked mon.noname-a con 0x7f726c06fe90 addr v1:172.21.15.84:6789/0 2024-03-29T10:35:53.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.068+0000 7f727212b640 10 monclient(hunting): picked mon.noname-b con 0x7f726c143c50 addr v1:172.21.15.84:6790/0 2024-03-29T10:35:53.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.068+0000 7f727212b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f726c072630 con 0x7f726c06fe90 2024-03-29T10:35:53.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.068+0000 7f727212b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f726c071410 con 0x7f726c0708a0 2024-03-29T10:35:53.054 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.068+0000 7f727212b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f726c0afaa0 con 0x7f726c143c50 2024-03-29T10:35:53.054 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.068+0000 7f727212b640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:53.054 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.068+0000 7f727212b640 10 monclient(hunting): authenticate will time out at 855.331700s 2024-03-29T10:35:53.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.072+0000 7f726bfff640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f726c143c50 0x7f726c14b8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:46295/0 (socket says 172.21.15.84:46295) 2024-03-29T10:35:53.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.072+0000 7f726bfff640 1 -- 172.21.15.84:0/410966936 learned_addr learned my addr 172.21.15.84:0/410966936 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:35:53.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.073+0000 7f726a7fc640 1 -- 172.21.15.84:0/410966936 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1479157845 0 0) 0x7f726c0afaa0 con 0x7f726c143c50 2024-03-29T10:35:53.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.073+0000 7f726a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:35:53.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.073+0000 7f726a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:53.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.073+0000 7f726a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:53.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.073+0000 7f726a7fc640 10 monclient(hunting): my global_id is 4268 2024-03-29T10:35:53.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.073+0000 7f726a7fc640 1 -- 172.21.15.84:0/410966936 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7254003520 con 0x7f726c143c50 2024-03-29T10:35:53.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.073+0000 7f726a7fc640 1 -- 172.21.15.84:0/410966936 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3575804690 0 0) 0x7f7254003520 con 0x7f726c143c50 2024-03-29T10:35:53.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.073+0000 7f726a7fc640 1 -- 172.21.15.84:0/410966936 >> v1:172.21.15.86:6789/0 conn(0x7f726c0708a0 legacy=0x7f726c0af390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:35:53.060 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.074+0000 7f726a7fc640 1 -- 172.21.15.84:0/410966936 >> v1:172.21.15.84:6789/0 conn(0x7f726c06fe90 legacy=0x7f726c143540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:53.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.074+0000 7f726a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:35:53.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.074+0000 7f726a7fc640 1 monclient: found mon.noname-b 2024-03-29T10:35:53.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.074+0000 7f726a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:53.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.074+0000 7f726a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:35:53.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.074+0000 7f726a7fc640 1 -- 172.21.15.84:0/410966936 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f726c0ac8b0 con 0x7f726c143c50 2024-03-29T10:35:53.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.074+0000 7f726a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:35:53.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.074+0000 7f726a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:35:53.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.074+0000 7f726a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:53.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.074+0000 7f726a7fc640 1 -- 172.21.15.84:0/410966936 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f7260001670 con 0x7f726c143c50 2024-03-29T10:35:53.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.074+0000 7f726a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:53.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.074+0000 7f726a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:35:53.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.074+0000 7f726a7fc640 10 monclient: dump: 2024-03-29T10:35:53.061 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:53.062 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:53.062 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:53.062 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:53.062 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:53.062 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:53.062 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:53.062 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:53.062 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:53.062 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:53.063 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.077+0000 7f727212b640 5 monclient: authenticate success, global_id 4268 2024-03-29T10:35:53.063 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.077+0000 7f726a7fc640 1 -- 172.21.15.84:0/410966936 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7260001900 con 0x7f726c143c50 2024-03-29T10:35:53.063 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.077+0000 7f726a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:53.063 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.077+0000 7f726a7fc640 1 -- 172.21.15.84:0/410966936 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f7260001cd0 con 0x7f726c143c50 2024-03-29T10:35:53.063 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.077+0000 7f726a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:53.064 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.077+0000 7f726a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:35:53.064 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.077+0000 7f726a7fc640 10 monclient: dump: 2024-03-29T10:35:53.064 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:53.064 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:53.064 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:53.064 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:53.064 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:53.064 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:53.064 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:53.064 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:53.064 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:53.064 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:53.064 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.077+0000 7f727212b640 10 monclient: get_monmap_and_config success 2024-03-29T10:35:53.064 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.077+0000 7f727212b640 4 set_mon_vals no callback set 2024-03-29T10:35:53.064 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.078+0000 7f727212b640 10 monclient: shutdown 2024-03-29T10:35:53.065 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.078+0000 7f727212b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:53.065 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.078+0000 7f727212b640 1 -- 172.21.15.84:0/410966936 >> v1:172.21.15.84:6790/0 conn(0x7f726c143c50 legacy=0x7f726c14b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:53.065 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.078+0000 7f727212b640 1 -- 172.21.15.84:0/410966936 shutdown_connections 2024-03-29T10:35:53.065 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.078+0000 7f727212b640 1 -- 172.21.15.84:0/410966936 >> 172.21.15.84:0/410966936 conn(0x7f726c0675b0 msgr2=0x7f726c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:53.068 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.083+0000 7f727212b640 1 -- 172.21.15.84:0/410966936 shutdown_connections 2024-03-29T10:35:53.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.084+0000 7f727212b640 1 -- 172.21.15.84:0/410966936 wait complete. 2024-03-29T10:35:53.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.085+0000 7f727212b640 10 monclient: build_initial_monmap 2024-03-29T10:35:53.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.085+0000 7f727212b640 10 monclient: monmap: 2024-03-29T10:35:53.070 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:35:53.070 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:35:53.071 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:35:53.071 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:35:53.071 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:35:53.071 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:53.071 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:35:53.071 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:35:53.071 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:35:53.071 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:53.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.085+0000 7f727212b640 1 librados: starting msgr at 2024-03-29T10:35:53.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.085+0000 7f727212b640 1 librados: starting objecter 2024-03-29T10:35:53.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.086+0000 7f727212b640 1 Processor -- start 2024-03-29T10:35:53.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.086+0000 7f727212b640 1 -- start start 2024-03-29T10:35:53.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.086+0000 7f727212b640 1 librados: setting wanted keys 2024-03-29T10:35:53.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.086+0000 7f727212b640 1 librados: calling monclient init 2024-03-29T10:35:53.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.086+0000 7f727212b640 10 monclient: init 2024-03-29T10:35:53.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.087+0000 7f727212b640 10 monclient: _reopen_session rank -1 2024-03-29T10:35:53.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.087+0000 7f727212b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:35:53.073 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.087+0000 7f727212b640 10 monclient(hunting): picked mon.noname-c con 0x7f726c06fe90 addr v1:172.21.15.84:6790/0 2024-03-29T10:35:53.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.087+0000 7f727212b640 10 monclient(hunting): picked mon.noname-b con 0x7f726c0708a0 addr v1:172.21.15.86:6789/0 2024-03-29T10:35:53.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.087+0000 7f727212b640 10 monclient(hunting): picked mon.noname-a con 0x7f726c143c50 addr v1:172.21.15.84:6789/0 2024-03-29T10:35:53.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.087+0000 7f727212b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f726c0ab280 con 0x7f726c143c50 2024-03-29T10:35:53.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.087+0000 7f727212b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f726c0ab4d0 con 0x7f726c0708a0 2024-03-29T10:35:53.075 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.087+0000 7f727212b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f726c0ab7b0 con 0x7f726c06fe90 2024-03-29T10:35:53.075 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.087+0000 7f727212b640 10 monclient(hunting): _renew_subs 2024-03-29T10:35:53.075 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.087+0000 7f727212b640 10 monclient(hunting): authenticate will time out at 855.350652s 2024-03-29T10:35:53.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.093+0000 7f726affd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f726c0708a0 0x7f726c0a9cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:60703/0 (socket says 172.21.15.84:60703) 2024-03-29T10:35:53.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.093+0000 7f726affd640 1 -- 172.21.15.84:0/1190035948 learned_addr learned my addr 172.21.15.84:0/1190035948 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:35:53.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.094+0000 7f7253fff640 10 client.?.objecter ms_handle_connect 0x7f726c0708a0 2024-03-29T10:35:53.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.094+0000 7f7253fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:53.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.094+0000 7f7253fff640 10 client.?.objecter ms_handle_connect 0x7f726c06fe90 2024-03-29T10:35:53.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.094+0000 7f7253fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:35:53.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.094+0000 7f7253fff640 1 -- 172.21.15.84:0/1190035948 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1069704985 0 0) 0x7f726c0ab7b0 con 0x7f726c06fe90 2024-03-29T10:35:53.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.094+0000 7f7253fff640 10 monclient(hunting): _negotiate 2024-03-29T10:35:53.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.094+0000 7f7253fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:53.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.094+0000 7f7253fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:53.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.094+0000 7f7253fff640 10 monclient(hunting): my global_id is 4271 2024-03-29T10:35:53.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.094+0000 7f7253fff640 1 -- 172.21.15.84:0/1190035948 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7258004eb0 con 0x7f726c06fe90 2024-03-29T10:35:53.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.094+0000 7f7253fff640 1 -- 172.21.15.84:0/1190035948 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2880159448 0 0) 0x7f726c0ab4d0 con 0x7f726c0708a0 2024-03-29T10:35:53.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.094+0000 7f7253fff640 10 monclient(hunting): _negotiate 2024-03-29T10:35:53.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.094+0000 7f7253fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:35:53.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.094+0000 7f7253fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:35:53.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.094+0000 7f7253fff640 10 monclient(hunting): my global_id is 4264 2024-03-29T10:35:53.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.094+0000 7f7253fff640 1 -- 172.21.15.84:0/1190035948 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f726c0ab7b0 con 0x7f726c0708a0 2024-03-29T10:35:53.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.095+0000 7f7253fff640 1 -- 172.21.15.84:0/1190035948 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1357795044 0 0) 0x7f7258004eb0 con 0x7f726c06fe90 2024-03-29T10:35:53.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.095+0000 7f7253fff640 1 -- 172.21.15.84:0/1190035948 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f726c0ab4d0 con 0x7f726c06fe90 2024-03-29T10:35:53.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.095+0000 7f7253fff640 1 -- 172.21.15.84:0/1190035948 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f725c001bf0 con 0x7f726c06fe90 2024-03-29T10:35:53.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.095+0000 7f7253fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:53.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.095+0000 7f7253fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:35:53.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.095+0000 7f7253fff640 10 monclient(hunting): dump: 2024-03-29T10:35:53.086 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:53.086 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:53.086 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:53.086 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:53.086 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:53.086 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:53.086 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:53.086 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:53.086 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:53.086 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:53.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.095+0000 7f7253fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:35:53.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.095+0000 7f7253fff640 1 -- 172.21.15.84:0/1190035948 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3087604547 0 0) 0x7f726c0ab4d0 con 0x7f726c06fe90 2024-03-29T10:35:53.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.095+0000 7f7253fff640 1 -- 172.21.15.84:0/1190035948 >> v1:172.21.15.86:6789/0 conn(0x7f726c0708a0 legacy=0x7f726c0a9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:53.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.095+0000 7f7253fff640 1 -- 172.21.15.84:0/1190035948 >> v1:172.21.15.84:6789/0 conn(0x7f726c143c50 legacy=0x7f726c1e04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:53.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.095+0000 7f7253fff640 10 monclient: _finish_hunting 0 2024-03-29T10:35:53.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.095+0000 7f7253fff640 1 monclient: found mon.c 2024-03-29T10:35:53.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.095+0000 7f7253fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:35:53.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.095+0000 7f7253fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:35:53.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.095+0000 7f7253fff640 1 -- 172.21.15.84:0/1190035948 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f726c0aba90 con 0x7f726c06fe90 2024-03-29T10:35:53.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.095+0000 7f7253fff640 10 monclient: _finish_auth 0 2024-03-29T10:35:53.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.095+0000 7f7253fff640 10 monclient: _check_auth_tickets 2024-03-29T10:35:53.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.095+0000 7f7253fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:35:53.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.095+0000 7f7253fff640 1 -- 172.21.15.84:0/1190035948 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f725c002100 con 0x7f726c06fe90 2024-03-29T10:35:53.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.095+0000 7f7253fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:35:53.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.095+0000 7f7253fff640 1 -- 172.21.15.84:0/1190035948 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f725c003bb0 con 0x7f726c06fe90 2024-03-29T10:35:53.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.095+0000 7f7253fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:35:53.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.095+0000 7f7253fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:35:53.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.095+0000 7f7253fff640 10 monclient: dump: 2024-03-29T10:35:53.088 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:35:53.088 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:35:53.088 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:53.088 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:35:53.088 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:35:53.088 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:35:53.088 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:35:53.088 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:35:53.088 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:35:53.088 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:35:53.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.095+0000 7f72697fa640 4 set_mon_vals no callback set 2024-03-29T10:35:53.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.096+0000 7f727212b640 5 monclient: authenticate success, global_id 4271 2024-03-29T10:35:53.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.096+0000 7f727212b640 10 monclient: _renew_subs 2024-03-29T10:35:53.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.096+0000 7f727212b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:35:53.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.096+0000 7f727212b640 1 -- 172.21.15.84:0/1190035948 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f726c13f960 con 0x7f726c06fe90 2024-03-29T10:35:53.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.096+0000 7f727212b640 10 client.4271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:35:53.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.096+0000 7f727212b640 10 monclient: _renew_subs 2024-03-29T10:35:53.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.096+0000 7f727212b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:35:53.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.096+0000 7f727212b640 1 -- 172.21.15.84:0/1190035948 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f726c13f6f0 con 0x7f726c06fe90 2024-03-29T10:35:53.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.096+0000 7f727212b640 1 librados: init done 2024-03-29T10:35:53.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.098+0000 7f7253fff640 1 -- 172.21.15.84:0/1190035948 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f725c019b80 con 0x7f726c06fe90 2024-03-29T10:35:53.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.100+0000 7f7253fff640 1 -- 172.21.15.84:0/1190035948 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f725c083b80 con 0x7f726c06fe90 2024-03-29T10:35:53.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.100+0000 7f726affd640 10 monclient: get_auth_request con 0x7f7258071eb0 auth_method 0 2024-03-29T10:35:53.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.100+0000 7f7253fff640 10 client.4271.objecter ms_dispatch 0x7f726c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:35:53.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.101+0000 7f7253fff640 3 client.4271.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:35:53.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.101+0000 7f7253fff640 3 client.4271.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:35:53.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.101+0000 7f7253fff640 20 client.4271.objecter dump_active .. 0 homeless 2024-03-29T10:35:53.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.101+0000 7f727212b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:53.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.101+0000 7f7253fff640 10 client.4271.objecter ms_handle_connect 0x7f7258071eb0 2024-03-29T10:35:53.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.101+0000 7f727212b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:53.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.101+0000 7f727212b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:35:53.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.101+0000 7f727212b640 1 -- 172.21.15.84:0/1190035948 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f726c071e40 con 0x7f726c06fe90 2024-03-29T10:35:53.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.110+0000 7f7253fff640 1 -- 172.21.15.84:0/1190035948 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f725c04f290 con 0x7f726c06fe90 2024-03-29T10:35:53.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.110+0000 7f7253fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:35:53.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.110+0000 7f7253fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:35:53.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.439+0000 7f727212b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:53.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.439+0000 7f727212b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:53.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.439+0000 7f727212b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:35:53.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.439+0000 7f727212b640 1 -- 172.21.15.84:0/1190035948 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f726c13fba0 con 0x7f726c06fe90 2024-03-29T10:35:53.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.440+0000 7f7253fff640 1 -- 172.21.15.84:0/1190035948 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f726c13fba0 con 0x7f726c06fe90 2024-03-29T10:35:53.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.440+0000 7f7253fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:35:53.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.440+0000 7f7253fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:35:53.427 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:35:53.427 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:35:53.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.445+0000 7f727212b640 10 librados: watch_flush enter 2024-03-29T10:35:53.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.445+0000 7f727212b640 10 librados: watch_flush exit 2024-03-29T10:35:53.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.445+0000 7f727212b640 20 client.4271.objecter shutdown clearing up homeless session... 2024-03-29T10:35:53.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.445+0000 7f727212b640 10 client.4271.objecter successfully canceled tick 2024-03-29T10:35:53.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.445+0000 7f727212b640 1 -- 172.21.15.84:0/1190035948 >> v1:172.21.15.86:6816/3337225337 conn(0x7f7258071eb0 legacy=0x7f7258074370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:53.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.445+0000 7f727212b640 10 monclient: shutdown 2024-03-29T10:35:53.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.445+0000 7f727212b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:35:53.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.446+0000 7f727212b640 1 -- 172.21.15.84:0/1190035948 >> v1:172.21.15.84:6790/0 conn(0x7f726c06fe90 legacy=0x7f726c0a95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:35:53.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.446+0000 7f727212b640 1 -- 172.21.15.84:0/1190035948 shutdown_connections 2024-03-29T10:35:53.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.446+0000 7f727212b640 1 -- 172.21.15.84:0/1190035948 >> 172.21.15.84:0/1190035948 conn(0x7f726c0675b0 msgr2=0x7f726c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:35:53.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.450+0000 7f727212b640 1 -- 172.21.15.84:0/1190035948 shutdown_connections 2024-03-29T10:35:53.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.450+0000 7f727212b640 1 -- 172.21.15.84:0/1190035948 wait complete. 2024-03-29T10:35:53.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:35:53.451+0000 7f727212b640 1 librados: shutdown 2024-03-29T10:35:53.453 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:36:00.260 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:36:00.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.420+0000 7f8e3aa25640 10 monclient: get_monmap_and_config 2024-03-29T10:36:00.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.420+0000 7f8e3aa25640 10 monclient: build_initial_monmap 2024-03-29T10:36:00.405 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.420+0000 7f8e3aa25640 10 monclient: monmap: 2024-03-29T10:36:00.405 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:36:00.405 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:00.405 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:36:00.421536+0000 2024-03-29T10:36:00.406 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:36:00.421536+0000 2024-03-29T10:36:00.406 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:00.406 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:00.406 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:36:00.406 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:36:00.406 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:36:00.406 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:00.406 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.421+0000 7f8e3aa25640 1 Processor -- start 2024-03-29T10:36:00.407 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.421+0000 7f8e3aa25640 1 -- start start 2024-03-29T10:36:00.407 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.421+0000 7f8e3aa25640 10 monclient: init 2024-03-29T10:36:00.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.422+0000 7f8e3aa25640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:00.409 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.422+0000 7f8e3aa25640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:36:00.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.422+0000 7f8e3aa25640 10 monclient(hunting): picked mon.noname-b con 0x7f8e341457f0 addr v1:172.21.15.84:6790/0 2024-03-29T10:36:00.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.422+0000 7f8e3aa25640 10 monclient(hunting): picked mon.noname-a con 0x7f8e3413f650 addr v1:172.21.15.84:6789/0 2024-03-29T10:36:00.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.422+0000 7f8e3aa25640 10 monclient(hunting): picked mon.noname-c con 0x7f8e340a9d70 addr v1:172.21.15.86:6789/0 2024-03-29T10:36:00.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.422+0000 7f8e3aa25640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e340aaef0 con 0x7f8e3413f650 2024-03-29T10:36:00.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.422+0000 7f8e3aa25640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e34145bd0 con 0x7f8e340a9d70 2024-03-29T10:36:00.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.422+0000 7f8e33fff640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f8e341457f0 0x7f8e3413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:60052/0 (socket says 172.21.15.84:60052) 2024-03-29T10:36:00.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.422+0000 7f8e33fff640 1 -- 172.21.15.84:0/3054683975 learned_addr learned my addr 172.21.15.84:0/3054683975 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:36:00.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.422+0000 7f8e3aa25640 1 -- 172.21.15.84:0/3054683975 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e34145eb0 con 0x7f8e341457f0 2024-03-29T10:36:00.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e3aa25640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:00.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e3aa25640 10 monclient(hunting): authenticate will time out at 862.686129s 2024-03-29T10:36:00.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e32ffd640 1 -- 172.21.15.84:0/3054683975 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2825726529 0 0) 0x7f8e34145eb0 con 0x7f8e341457f0 2024-03-29T10:36:00.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e32ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:36:00.410 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:00.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:00.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e32ffd640 10 monclient(hunting): my global_id is 4274 2024-03-29T10:36:00.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e32ffd640 1 -- 172.21.15.84:0/3054683975 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e1c003520 con 0x7f8e341457f0 2024-03-29T10:36:00.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e32ffd640 1 -- 172.21.15.84:0/3054683975 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3363673563 0 0) 0x7f8e340aaef0 con 0x7f8e3413f650 2024-03-29T10:36:00.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e32ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:36:00.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:00.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:00.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e32ffd640 10 monclient(hunting): my global_id is 4272 2024-03-29T10:36:00.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e32ffd640 1 -- 172.21.15.84:0/3054683975 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e34145eb0 con 0x7f8e3413f650 2024-03-29T10:36:00.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e32ffd640 1 -- 172.21.15.84:0/3054683975 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 252362519 0 0) 0x7f8e1c003520 con 0x7f8e341457f0 2024-03-29T10:36:00.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e32ffd640 1 -- 172.21.15.84:0/3054683975 >> v1:172.21.15.86:6789/0 conn(0x7f8e340a9d70 legacy=0x7f8e3414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:36:00.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e32ffd640 1 -- 172.21.15.84:0/3054683975 >> v1:172.21.15.84:6789/0 conn(0x7f8e3413f650 legacy=0x7f8e340b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:00.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e32ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:36:00.411 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e32ffd640 1 monclient: found mon.noname-b 2024-03-29T10:36:00.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:00.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e32ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:36:00.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e32ffd640 1 -- 172.21.15.84:0/3054683975 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e34146340 con 0x7f8e341457f0 2024-03-29T10:36:00.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e32ffd640 10 monclient: _finish_auth 0 2024-03-29T10:36:00.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e32ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:36:00.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:00.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e3aa25640 5 monclient: authenticate success, global_id 4274 2024-03-29T10:36:00.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e3aa25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:00.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e32ffd640 1 -- 172.21.15.84:0/3054683975 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f8e2c0019d0 con 0x7f8e341457f0 2024-03-29T10:36:00.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:00.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e32ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:36:00.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e32ffd640 10 monclient: dump: 2024-03-29T10:36:00.412 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:00.412 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:00.412 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:00.413 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:00.413 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:00.413 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:00.413 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:00.413 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:00.413 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:00.413 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:00.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.423+0000 7f8e3aa25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:00.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.424+0000 7f8e32ffd640 1 -- 172.21.15.84:0/3054683975 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e2c001c20 con 0x7f8e341457f0 2024-03-29T10:36:00.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.424+0000 7f8e32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:00.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.424+0000 7f8e32ffd640 1 -- 172.21.15.84:0/3054683975 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f8e2c001fb0 con 0x7f8e341457f0 2024-03-29T10:36:00.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.424+0000 7f8e32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:00.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.424+0000 7f8e32ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:00.413 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.424+0000 7f8e32ffd640 10 monclient: dump: 2024-03-29T10:36:00.414 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:00.414 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:00.414 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:00.414 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:00.414 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:00.414 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:00.414 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:00.414 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:00.414 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:00.414 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:00.414 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.424+0000 7f8e3aa25640 10 monclient: get_monmap_and_config success 2024-03-29T10:36:00.414 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.424+0000 7f8e3aa25640 4 set_mon_vals no callback set 2024-03-29T10:36:00.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.425+0000 7f8e3aa25640 10 monclient: shutdown 2024-03-29T10:36:00.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.425+0000 7f8e3aa25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:00.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.425+0000 7f8e3aa25640 1 -- 172.21.15.84:0/3054683975 >> v1:172.21.15.84:6790/0 conn(0x7f8e341457f0 legacy=0x7f8e3413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:00.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.425+0000 7f8e3aa25640 1 -- 172.21.15.84:0/3054683975 shutdown_connections 2024-03-29T10:36:00.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.425+0000 7f8e3aa25640 1 -- 172.21.15.84:0/3054683975 >> 172.21.15.84:0/3054683975 conn(0x7f8e34066a00 msgr2=0x7f8e34066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:00.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.425+0000 7f8e3aa25640 1 -- 172.21.15.84:0/3054683975 shutdown_connections 2024-03-29T10:36:00.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.425+0000 7f8e3aa25640 1 -- 172.21.15.84:0/3054683975 wait complete. 2024-03-29T10:36:00.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.426+0000 7f8e3aa25640 10 monclient: build_initial_monmap 2024-03-29T10:36:00.420 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.426+0000 7f8e3aa25640 10 monclient: monmap: 2024-03-29T10:36:00.420 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:36:00.421 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:36:00.421 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:36:00.421 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:36:00.421 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:00.421 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:00.421 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:36:00.421 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:36:00.421 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:36:00.421 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:00.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.427+0000 7f8e3aa25640 1 librados: starting msgr at 2024-03-29T10:36:00.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.427+0000 7f8e3aa25640 1 librados: starting objecter 2024-03-29T10:36:00.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.427+0000 7f8e3aa25640 1 Processor -- start 2024-03-29T10:36:00.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.427+0000 7f8e3aa25640 1 -- start start 2024-03-29T10:36:00.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.427+0000 7f8e3aa25640 1 librados: setting wanted keys 2024-03-29T10:36:00.421 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.427+0000 7f8e3aa25640 1 librados: calling monclient init 2024-03-29T10:36:00.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.427+0000 7f8e3aa25640 10 monclient: init 2024-03-29T10:36:00.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.428+0000 7f8e3aa25640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:00.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.428+0000 7f8e3aa25640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:36:00.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.428+0000 7f8e3aa25640 10 monclient(hunting): picked mon.noname-a con 0x7f8e340a9d70 addr v1:172.21.15.84:6789/0 2024-03-29T10:36:00.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.428+0000 7f8e3aa25640 10 monclient(hunting): picked mon.noname-c con 0x7f8e3413f650 addr v1:172.21.15.84:6790/0 2024-03-29T10:36:00.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.428+0000 7f8e3aa25640 10 monclient(hunting): picked mon.noname-b con 0x7f8e341457f0 addr v1:172.21.15.86:6789/0 2024-03-29T10:36:00.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.428+0000 7f8e3aa25640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e341dfc30 con 0x7f8e340a9d70 2024-03-29T10:36:00.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.428+0000 7f8e337fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f8e3413f650 0x7f8e341dee10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:60200/0 (socket says 172.21.15.84:60200) 2024-03-29T10:36:00.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.428+0000 7f8e337fe640 1 -- 172.21.15.84:0/1240873185 learned_addr learned my addr 172.21.15.84:0/1240873185 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:36:00.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.428+0000 7f8e3aa25640 1 -- 172.21.15.84:0/1240873185 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e34141d00 con 0x7f8e341457f0 2024-03-29T10:36:00.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.428+0000 7f8e3aa25640 1 -- 172.21.15.84:0/1240873185 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e3406eda0 con 0x7f8e3413f650 2024-03-29T10:36:00.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.428+0000 7f8e3aa25640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:00.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.428+0000 7f8e3aa25640 10 monclient(hunting): authenticate will time out at 862.691904s 2024-03-29T10:36:00.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.428+0000 7f8e30ff9640 10 client.?.objecter ms_handle_connect 0x7f8e341457f0 2024-03-29T10:36:00.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.428+0000 7f8e30ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:00.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.428+0000 7f8e30ff9640 10 client.?.objecter ms_handle_connect 0x7f8e3413f650 2024-03-29T10:36:00.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.428+0000 7f8e30ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:00.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.429+0000 7f8e30ff9640 10 client.?.objecter ms_handle_connect 0x7f8e340a9d70 2024-03-29T10:36:00.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.429+0000 7f8e30ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:00.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.429+0000 7f8e30ff9640 1 -- 172.21.15.84:0/1240873185 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130532230 0 0) 0x7f8e34141d00 con 0x7f8e341457f0 2024-03-29T10:36:00.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.429+0000 7f8e30ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:36:00.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.429+0000 7f8e30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:00.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.429+0000 7f8e30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:00.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.429+0000 7f8e30ff9640 10 monclient(hunting): my global_id is 4267 2024-03-29T10:36:00.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.429+0000 7f8e30ff9640 1 -- 172.21.15.84:0/1240873185 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e20004eb0 con 0x7f8e341457f0 2024-03-29T10:36:00.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.429+0000 7f8e30ff9640 1 -- 172.21.15.84:0/1240873185 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3651777145 0 0) 0x7f8e341dfc30 con 0x7f8e340a9d70 2024-03-29T10:36:00.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.429+0000 7f8e30ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:36:00.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.429+0000 7f8e30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:00.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.429+0000 7f8e30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:00.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.429+0000 7f8e30ff9640 10 monclient(hunting): my global_id is 4275 2024-03-29T10:36:00.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.429+0000 7f8e30ff9640 1 -- 172.21.15.84:0/1240873185 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e34141d00 con 0x7f8e340a9d70 2024-03-29T10:36:00.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.429+0000 7f8e30ff9640 1 -- 172.21.15.84:0/1240873185 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1768105006 0 0) 0x7f8e3406eda0 con 0x7f8e3413f650 2024-03-29T10:36:00.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.429+0000 7f8e30ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:36:00.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.429+0000 7f8e30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:00.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.429+0000 7f8e30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:00.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.429+0000 7f8e30ff9640 10 monclient(hunting): my global_id is 4277 2024-03-29T10:36:00.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.429+0000 7f8e30ff9640 1 -- 172.21.15.84:0/1240873185 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e341dfc30 con 0x7f8e3413f650 2024-03-29T10:36:00.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.429+0000 7f8e30ff9640 1 -- 172.21.15.84:0/1240873185 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 753405298 0 0) 0x7f8e20004eb0 con 0x7f8e341457f0 2024-03-29T10:36:00.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.429+0000 7f8e30ff9640 1 -- 172.21.15.84:0/1240873185 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e3406eda0 con 0x7f8e341457f0 2024-03-29T10:36:00.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.429+0000 7f8e30ff9640 1 -- 172.21.15.84:0/1240873185 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f8e240017f0 con 0x7f8e341457f0 2024-03-29T10:36:00.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.429+0000 7f8e30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:00.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.430+0000 7f8e30ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:36:00.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.430+0000 7f8e30ff9640 10 monclient(hunting): dump: 2024-03-29T10:36:00.425 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:00.425 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:00.425 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:00.425 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:00.425 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:00.425 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:00.425 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:00.425 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:00.425 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:00.425 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:00.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.430+0000 7f8e30ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:36:00.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.430+0000 7f8e30ff9640 1 -- 172.21.15.84:0/1240873185 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3825539354 0 0) 0x7f8e34141d00 con 0x7f8e340a9d70 2024-03-29T10:36:00.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.430+0000 7f8e30ff9640 1 -- 172.21.15.84:0/1240873185 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e20004eb0 con 0x7f8e340a9d70 2024-03-29T10:36:00.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.430+0000 7f8e30ff9640 1 -- 172.21.15.84:0/1240873185 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f8e2c001f30 con 0x7f8e340a9d70 2024-03-29T10:36:00.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.430+0000 7f8e30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:00.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.430+0000 7f8e30ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:36:00.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.430+0000 7f8e30ff9640 10 monclient(hunting): dump: 2024-03-29T10:36:00.426 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:00.426 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:00.426 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:00.426 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:00.426 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:00.426 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:00.426 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:00.426 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:00.426 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:00.426 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:00.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.430+0000 7f8e30ff9640 1 -- 172.21.15.84:0/1240873185 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 631124400 0 0) 0x7f8e341dfc30 con 0x7f8e3413f650 2024-03-29T10:36:00.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.430+0000 7f8e30ff9640 1 -- 172.21.15.84:0/1240873185 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e34141d00 con 0x7f8e3413f650 2024-03-29T10:36:00.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.430+0000 7f8e30ff9640 1 -- 172.21.15.84:0/1240873185 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f8e28001930 con 0x7f8e3413f650 2024-03-29T10:36:00.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.430+0000 7f8e30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:00.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.430+0000 7f8e30ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:00.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.430+0000 7f8e30ff9640 10 monclient(hunting): dump: 2024-03-29T10:36:00.427 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:00.427 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:00.427 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:00.427 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:00.427 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:00.427 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:00.428 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:00.428 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:00.428 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:00.428 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:00.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.431+0000 7f8e30ff9640 1 -- 172.21.15.84:0/1240873185 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 683819543 0 0) 0x7f8e34141d00 con 0x7f8e3413f650 2024-03-29T10:36:00.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.431+0000 7f8e30ff9640 1 -- 172.21.15.84:0/1240873185 >> v1:172.21.15.86:6789/0 conn(0x7f8e341457f0 legacy=0x7f8e341df520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:00.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.431+0000 7f8e30ff9640 1 -- 172.21.15.84:0/1240873185 >> v1:172.21.15.84:6789/0 conn(0x7f8e340a9d70 legacy=0x7f8e341425b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:00.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.431+0000 7f8e30ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:36:00.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.431+0000 7f8e30ff9640 1 monclient: found mon.c 2024-03-29T10:36:00.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.431+0000 7f8e30ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:00.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.431+0000 7f8e30ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:36:00.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.431+0000 7f8e30ff9640 1 -- 172.21.15.84:0/1240873185 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e3406f080 con 0x7f8e3413f650 2024-03-29T10:36:00.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.431+0000 7f8e30ff9640 10 monclient: _finish_auth 0 2024-03-29T10:36:00.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.431+0000 7f8e30ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:36:00.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.431+0000 7f8e30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:00.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.431+0000 7f8e3aa25640 5 monclient: authenticate success, global_id 4277 2024-03-29T10:36:00.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.431+0000 7f8e3aa25640 10 monclient: _renew_subs 2024-03-29T10:36:00.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.431+0000 7f8e3aa25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:36:00.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.431+0000 7f8e3aa25640 1 -- 172.21.15.84:0/1240873185 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e34071f80 con 0x7f8e3413f650 2024-03-29T10:36:00.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.431+0000 7f8e3aa25640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:36:00.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.431+0000 7f8e3aa25640 10 monclient: _renew_subs 2024-03-29T10:36:00.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.431+0000 7f8e3aa25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:36:00.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.431+0000 7f8e3aa25640 1 -- 172.21.15.84:0/1240873185 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e34142ea0 con 0x7f8e3413f650 2024-03-29T10:36:00.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.431+0000 7f8e3aa25640 1 librados: init done 2024-03-29T10:36:00.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.433+0000 7f8e30ff9640 1 -- 172.21.15.84:0/1240873185 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e28003610 con 0x7f8e3413f650 2024-03-29T10:36:00.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.433+0000 7f8e30ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:00.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.433+0000 7f8e30ff9640 1 -- 172.21.15.84:0/1240873185 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f8e28003880 con 0x7f8e3413f650 2024-03-29T10:36:00.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.433+0000 7f8e30ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:00.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.433+0000 7f8e30ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:00.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.433+0000 7f8e30ff9640 10 monclient: dump: 2024-03-29T10:36:00.430 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:00.430 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:00.430 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:00.430 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:00.430 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:00.430 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:00.430 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:00.430 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:00.430 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:00.430 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:00.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.433+0000 7f8e31ffb640 4 set_mon_vals no callback set 2024-03-29T10:36:00.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.433+0000 7f8e3aa25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:00.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.434+0000 7f8e3aa25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:00.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.434+0000 7f8e3aa25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:36:00.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.434+0000 7f8e3aa25640 1 -- 172.21.15.84:0/1240873185 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e14005080 con 0x7f8e3413f650 2024-03-29T10:36:00.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.435+0000 7f8e30ff9640 1 -- 172.21.15.84:0/1240873185 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f8e28003a60 con 0x7f8e3413f650 2024-03-29T10:36:00.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.436+0000 7f8e33fff640 10 monclient: get_auth_request con 0x7f8e2006e0f0 auth_method 0 2024-03-29T10:36:00.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.436+0000 7f8e30ff9640 1 -- 172.21.15.84:0/1240873185 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f8e28084870 con 0x7f8e3413f650 2024-03-29T10:36:00.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.436+0000 7f8e30ff9640 10 client.4277.objecter ms_dispatch 0x7f8e34000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:36:00.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.436+0000 7f8e30ff9640 3 client.4277.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:36:00.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.436+0000 7f8e30ff9640 3 client.4277.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:36:00.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.436+0000 7f8e30ff9640 20 client.4277.objecter dump_active .. 0 homeless 2024-03-29T10:36:00.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.436+0000 7f8e30ff9640 10 client.4277.objecter ms_handle_connect 0x7f8e2006e0f0 2024-03-29T10:36:00.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.445+0000 7f8e30ff9640 1 -- 172.21.15.84:0/1240873185 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8e28050150 con 0x7f8e3413f650 2024-03-29T10:36:00.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.445+0000 7f8e30ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:00.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.445+0000 7f8e30ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:36:00.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.800+0000 7f8e3aa25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:00.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.800+0000 7f8e3aa25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:00.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.800+0000 7f8e3aa25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:36:00.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.800+0000 7f8e3aa25640 1 -- 172.21.15.84:0/1240873185 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e14005330 con 0x7f8e3413f650 2024-03-29T10:36:00.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.801+0000 7f8e30ff9640 1 -- 172.21.15.84:0/1240873185 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f8e28052080 con 0x7f8e3413f650 2024-03-29T10:36:00.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.801+0000 7f8e30ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:00.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.801+0000 7f8e30ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:36:00.786 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:36:00.787 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:36:00.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.804+0000 7f8e3aa25640 10 librados: watch_flush enter 2024-03-29T10:36:00.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.804+0000 7f8e3aa25640 10 librados: watch_flush exit 2024-03-29T10:36:00.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.804+0000 7f8e3aa25640 20 client.4277.objecter shutdown clearing up homeless session... 2024-03-29T10:36:00.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.804+0000 7f8e3aa25640 10 client.4277.objecter successfully canceled tick 2024-03-29T10:36:00.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.804+0000 7f8e3aa25640 1 -- 172.21.15.84:0/1240873185 >> v1:172.21.15.86:6816/3337225337 conn(0x7f8e2006e0f0 legacy=0x7f8e200705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:00.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.804+0000 7f8e3aa25640 10 monclient: shutdown 2024-03-29T10:36:00.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.804+0000 7f8e3aa25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:00.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.805+0000 7f8e3aa25640 1 -- 172.21.15.84:0/1240873185 >> v1:172.21.15.84:6790/0 conn(0x7f8e3413f650 legacy=0x7f8e341dee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:00.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.805+0000 7f8e3aa25640 1 -- 172.21.15.84:0/1240873185 shutdown_connections 2024-03-29T10:36:00.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.805+0000 7f8e3aa25640 1 -- 172.21.15.84:0/1240873185 >> 172.21.15.84:0/1240873185 conn(0x7f8e34066a00 msgr2=0x7f8e34066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:00.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.805+0000 7f8e3aa25640 1 -- 172.21.15.84:0/1240873185 shutdown_connections 2024-03-29T10:36:00.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.805+0000 7f8e3aa25640 1 -- 172.21.15.84:0/1240873185 wait complete. 2024-03-29T10:36:00.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:00.805+0000 7f8e3aa25640 1 librados: shutdown 2024-03-29T10:36:00.805 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:36:07.611 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:36:07.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.776+0000 7f1c8d94d640 10 monclient: get_monmap_and_config 2024-03-29T10:36:07.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.776+0000 7f1c8d94d640 10 monclient: build_initial_monmap 2024-03-29T10:36:07.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.776+0000 7f1c8d94d640 10 monclient: monmap: 2024-03-29T10:36:07.761 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:36:07.761 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:07.761 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:36:07.777059+0000 2024-03-29T10:36:07.761 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:36:07.777059+0000 2024-03-29T10:36:07.761 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:07.761 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:07.761 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:36:07.762 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:36:07.762 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:36:07.762 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:07.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.777+0000 7f1c8d94d640 1 Processor -- start 2024-03-29T10:36:07.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.777+0000 7f1c8d94d640 1 -- start start 2024-03-29T10:36:07.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.777+0000 7f1c8d94d640 10 monclient: init 2024-03-29T10:36:07.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.778+0000 7f1c8d94d640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:07.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.778+0000 7f1c8d94d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:36:07.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.778+0000 7f1c8d94d640 10 monclient(hunting): picked mon.noname-a con 0x7f1c88149c50 addr v1:172.21.15.84:6789/0 2024-03-29T10:36:07.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.778+0000 7f1c8d94d640 10 monclient(hunting): picked mon.noname-b con 0x7f1c880a9cc0 addr v1:172.21.15.84:6790/0 2024-03-29T10:36:07.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.778+0000 7f1c8d94d640 10 monclient(hunting): picked mon.noname-c con 0x7f1c88143d00 addr v1:172.21.15.86:6789/0 2024-03-29T10:36:07.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.778+0000 7f1c8d94d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c880ab9c0 con 0x7f1c88149c50 2024-03-29T10:36:07.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.778+0000 7f1c8d94d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c880abc10 con 0x7f1c88143d00 2024-03-29T10:36:07.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.778+0000 7f1c8d94d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c880aad20 con 0x7f1c880a9cc0 2024-03-29T10:36:07.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.778+0000 7f1c8d94d640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:07.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.778+0000 7f1c8d94d640 10 monclient(hunting): authenticate will time out at 870.041795s 2024-03-29T10:36:07.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.779+0000 7f1c877fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f1c88143d00 0x7f1c88151d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:36296/0 (socket says 172.21.15.84:36296) 2024-03-29T10:36:07.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.779+0000 7f1c877fe640 1 -- 172.21.15.84:0/3408977188 learned_addr learned my addr 172.21.15.84:0/3408977188 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:36:07.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.779+0000 7f1c85ffb640 1 -- 172.21.15.84:0/3408977188 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1459876910 0 0) 0x7f1c880abc10 con 0x7f1c88143d00 2024-03-29T10:36:07.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.779+0000 7f1c85ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:36:07.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.779+0000 7f1c85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:07.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.779+0000 7f1c85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:07.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.779+0000 7f1c85ffb640 10 monclient(hunting): my global_id is 4270 2024-03-29T10:36:07.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.779+0000 7f1c85ffb640 1 -- 172.21.15.84:0/3408977188 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c70003520 con 0x7f1c88143d00 2024-03-29T10:36:07.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.780+0000 7f1c85ffb640 1 -- 172.21.15.84:0/3408977188 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1236113707 0 0) 0x7f1c70003520 con 0x7f1c88143d00 2024-03-29T10:36:07.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.780+0000 7f1c85ffb640 1 -- 172.21.15.84:0/3408977188 >> v1:172.21.15.84:6790/0 conn(0x7f1c880a9cc0 legacy=0x7f1c881435f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:07.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.780+0000 7f1c85ffb640 1 -- 172.21.15.84:0/3408977188 >> v1:172.21.15.84:6789/0 conn(0x7f1c88149c50 legacy=0x7f1c880ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:07.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.780+0000 7f1c85ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:36:07.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.780+0000 7f1c85ffb640 1 monclient: found mon.noname-c 2024-03-29T10:36:07.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.780+0000 7f1c85ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:07.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.780+0000 7f1c85ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-29T10:36:07.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.780+0000 7f1c85ffb640 1 -- 172.21.15.84:0/3408977188 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c880aa980 con 0x7f1c88143d00 2024-03-29T10:36:07.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.780+0000 7f1c85ffb640 10 monclient: _finish_auth 0 2024-03-29T10:36:07.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.780+0000 7f1c85ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:36:07.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.780+0000 7f1c85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:07.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.780+0000 7f1c85ffb640 1 -- 172.21.15.84:0/3408977188 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f1c78001690 con 0x7f1c88143d00 2024-03-29T10:36:07.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.780+0000 7f1c85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:07.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.780+0000 7f1c85ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:36:07.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.780+0000 7f1c85ffb640 10 monclient: dump: 2024-03-29T10:36:07.770 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:07.771 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:07.771 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:07.771 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:07.771 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:07.771 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:07.771 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:07.771 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:07.771 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:07.771 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:07.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.780+0000 7f1c8d94d640 5 monclient: authenticate success, global_id 4270 2024-03-29T10:36:07.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.780+0000 7f1c8d94d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:07.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.781+0000 7f1c85ffb640 1 -- 172.21.15.84:0/3408977188 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c780017f0 con 0x7f1c88143d00 2024-03-29T10:36:07.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.781+0000 7f1c85ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:07.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.781+0000 7f1c85ffb640 1 -- 172.21.15.84:0/3408977188 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f1c78001c00 con 0x7f1c88143d00 2024-03-29T10:36:07.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.781+0000 7f1c85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:07.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.781+0000 7f1c85ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:07.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.781+0000 7f1c85ffb640 10 monclient: dump: 2024-03-29T10:36:07.772 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:07.772 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:07.772 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:07.772 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:07.772 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:07.772 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:07.772 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:07.772 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:07.772 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:07.772 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:07.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.781+0000 7f1c8d94d640 10 monclient: get_monmap_and_config success 2024-03-29T10:36:07.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.781+0000 7f1c8d94d640 4 set_mon_vals no callback set 2024-03-29T10:36:07.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.781+0000 7f1c8d94d640 10 monclient: shutdown 2024-03-29T10:36:07.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.781+0000 7f1c8d94d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:07.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.781+0000 7f1c8d94d640 1 -- 172.21.15.84:0/3408977188 >> v1:172.21.15.86:6789/0 conn(0x7f1c88143d00 legacy=0x7f1c88151d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:07.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.782+0000 7f1c8d94d640 1 -- 172.21.15.84:0/3408977188 shutdown_connections 2024-03-29T10:36:07.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.782+0000 7f1c8d94d640 1 -- 172.21.15.84:0/3408977188 >> 172.21.15.84:0/3408977188 conn(0x7f1c880b3c50 msgr2=0x7f1c880b4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:07.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.782+0000 7f1c8d94d640 1 -- 172.21.15.84:0/3408977188 shutdown_connections 2024-03-29T10:36:07.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.782+0000 7f1c8d94d640 1 -- 172.21.15.84:0/3408977188 wait complete. 2024-03-29T10:36:07.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.783+0000 7f1c8d94d640 10 monclient: build_initial_monmap 2024-03-29T10:36:07.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.783+0000 7f1c8d94d640 10 monclient: monmap: 2024-03-29T10:36:07.773 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:36:07.773 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:36:07.773 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:36:07.773 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:36:07.774 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:07.774 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:07.774 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:36:07.774 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:36:07.774 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:36:07.774 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:07.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.783+0000 7f1c8d94d640 1 librados: starting msgr at 2024-03-29T10:36:07.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.783+0000 7f1c8d94d640 1 librados: starting objecter 2024-03-29T10:36:07.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.783+0000 7f1c8d94d640 1 Processor -- start 2024-03-29T10:36:07.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.784+0000 7f1c8d94d640 1 -- start start 2024-03-29T10:36:07.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.784+0000 7f1c8d94d640 1 librados: setting wanted keys 2024-03-29T10:36:07.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.784+0000 7f1c8d94d640 1 librados: calling monclient init 2024-03-29T10:36:07.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.784+0000 7f1c8d94d640 10 monclient: init 2024-03-29T10:36:07.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.785+0000 7f1c8d94d640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:07.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.785+0000 7f1c8d94d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:36:07.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.785+0000 7f1c8d94d640 10 monclient(hunting): picked mon.noname-c con 0x7f1c880a9cc0 addr v1:172.21.15.84:6790/0 2024-03-29T10:36:07.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.785+0000 7f1c8d94d640 10 monclient(hunting): picked mon.noname-a con 0x7f1c88143d00 addr v1:172.21.15.84:6789/0 2024-03-29T10:36:07.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.785+0000 7f1c8d94d640 10 monclient(hunting): picked mon.noname-b con 0x7f1c88149c50 addr v1:172.21.15.86:6789/0 2024-03-29T10:36:07.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.785+0000 7f1c8d94d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c881dfb00 con 0x7f1c88143d00 2024-03-29T10:36:07.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.785+0000 7f1c8d94d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c8813d390 con 0x7f1c88149c50 2024-03-29T10:36:07.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.785+0000 7f1c8d94d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c881deb60 con 0x7f1c880a9cc0 2024-03-29T10:36:07.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.785+0000 7f1c877fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f1c88149c50 0x7f1c8813d790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:39150/0 (socket says 172.21.15.84:39150) 2024-03-29T10:36:07.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.785+0000 7f1c877fe640 1 -- 172.21.15.84:0/2587763120 learned_addr learned my addr 172.21.15.84:0/2587763120 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:36:07.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.785+0000 7f1c8d94d640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:07.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.785+0000 7f1c8d94d640 10 monclient(hunting): authenticate will time out at 870.048689s 2024-03-29T10:36:07.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.785+0000 7f1c6ffff640 10 client.?.objecter ms_handle_connect 0x7f1c88143d00 2024-03-29T10:36:07.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.785+0000 7f1c6ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:07.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.785+0000 7f1c6ffff640 10 client.?.objecter ms_handle_connect 0x7f1c88149c50 2024-03-29T10:36:07.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.785+0000 7f1c6ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:07.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.785+0000 7f1c6ffff640 10 client.?.objecter ms_handle_connect 0x7f1c880a9cc0 2024-03-29T10:36:07.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.785+0000 7f1c6ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:07.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.786+0000 7f1c6ffff640 1 -- 172.21.15.84:0/2587763120 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 499088109 0 0) 0x7f1c8813d390 con 0x7f1c88149c50 2024-03-29T10:36:07.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.786+0000 7f1c6ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:36:07.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.786+0000 7f1c6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:07.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.786+0000 7f1c6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:07.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.786+0000 7f1c6ffff640 10 monclient(hunting): my global_id is 4273 2024-03-29T10:36:07.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.786+0000 7f1c6ffff640 1 -- 172.21.15.84:0/2587763120 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c74004eb0 con 0x7f1c88149c50 2024-03-29T10:36:07.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.786+0000 7f1c6ffff640 1 -- 172.21.15.84:0/2587763120 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 138442620 0 0) 0x7f1c881deb60 con 0x7f1c880a9cc0 2024-03-29T10:36:07.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.786+0000 7f1c6ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:36:07.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.786+0000 7f1c6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:07.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.786+0000 7f1c6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:07.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.786+0000 7f1c6ffff640 10 monclient(hunting): my global_id is 4283 2024-03-29T10:36:07.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.786+0000 7f1c6ffff640 1 -- 172.21.15.84:0/2587763120 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c8813d390 con 0x7f1c880a9cc0 2024-03-29T10:36:07.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.786+0000 7f1c6ffff640 1 -- 172.21.15.84:0/2587763120 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590363950 0 0) 0x7f1c881dfb00 con 0x7f1c88143d00 2024-03-29T10:36:07.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.786+0000 7f1c6ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:36:07.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.786+0000 7f1c6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:07.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.786+0000 7f1c6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:07.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.786+0000 7f1c6ffff640 10 monclient(hunting): my global_id is 4281 2024-03-29T10:36:07.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.786+0000 7f1c6ffff640 1 -- 172.21.15.84:0/2587763120 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c881deb60 con 0x7f1c88143d00 2024-03-29T10:36:07.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.787+0000 7f1c6ffff640 1 -- 172.21.15.84:0/2587763120 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 374242841 0 0) 0x7f1c74004eb0 con 0x7f1c88149c50 2024-03-29T10:36:07.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.787+0000 7f1c6ffff640 1 -- 172.21.15.84:0/2587763120 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c881dfb00 con 0x7f1c88149c50 2024-03-29T10:36:07.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.787+0000 7f1c6ffff640 1 -- 172.21.15.84:0/2587763120 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f1c78001bc0 con 0x7f1c88149c50 2024-03-29T10:36:07.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.787+0000 7f1c6ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:07.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.787+0000 7f1c6ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:36:07.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.787+0000 7f1c6ffff640 10 monclient(hunting): dump: 2024-03-29T10:36:07.785 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:07.786 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:07.786 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:07.786 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:07.786 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:07.786 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:07.786 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:07.786 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:07.786 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:07.786 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:07.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.787+0000 7f1c6ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:36:07.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.787+0000 7f1c6ffff640 1 -- 172.21.15.84:0/2587763120 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2772981333 0 0) 0x7f1c8813d390 con 0x7f1c880a9cc0 2024-03-29T10:36:07.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.787+0000 7f1c6ffff640 1 -- 172.21.15.84:0/2587763120 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c74004eb0 con 0x7f1c880a9cc0 2024-03-29T10:36:07.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.787+0000 7f1c6ffff640 1 -- 172.21.15.84:0/2587763120 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f1c80001a40 con 0x7f1c880a9cc0 2024-03-29T10:36:07.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.787+0000 7f1c6ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:07.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.787+0000 7f1c6ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:07.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.787+0000 7f1c6ffff640 10 monclient(hunting): dump: 2024-03-29T10:36:07.787 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:07.787 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:07.787 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:07.787 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:07.787 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:07.787 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:07.787 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:07.787 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:07.787 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:07.787 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:07.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.787+0000 7f1c6ffff640 1 -- 172.21.15.84:0/2587763120 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2976010547 0 0) 0x7f1c881dfb00 con 0x7f1c88149c50 2024-03-29T10:36:07.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.788+0000 7f1c6ffff640 1 -- 172.21.15.84:0/2587763120 >> v1:172.21.15.84:6790/0 conn(0x7f1c880a9cc0 legacy=0x7f1c881df360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:07.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.788+0000 7f1c6ffff640 1 -- 172.21.15.84:0/2587763120 >> v1:172.21.15.84:6789/0 conn(0x7f1c88143d00 legacy=0x7f1c881d8810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:07.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.788+0000 7f1c6ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:36:07.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.788+0000 7f1c6ffff640 1 monclient: found mon.b 2024-03-29T10:36:07.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.788+0000 7f1c6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:07.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.788+0000 7f1c6ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:36:07.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.788+0000 7f1c6ffff640 1 -- 172.21.15.84:0/2587763120 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c881dee40 con 0x7f1c88149c50 2024-03-29T10:36:07.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.788+0000 7f1c6ffff640 10 monclient: _finish_auth 0 2024-03-29T10:36:07.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.788+0000 7f1c6ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:36:07.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.788+0000 7f1c6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:07.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.788+0000 7f1c8d94d640 5 monclient: authenticate success, global_id 4273 2024-03-29T10:36:07.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.788+0000 7f1c8d94d640 10 monclient: _renew_subs 2024-03-29T10:36:07.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.788+0000 7f1c8d94d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:36:07.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.788+0000 7f1c8d94d640 1 -- 172.21.15.84:0/2587763120 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c881d8f20 con 0x7f1c88149c50 2024-03-29T10:36:07.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.788+0000 7f1c8d94d640 10 client.4273.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:36:07.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.788+0000 7f1c8d94d640 10 monclient: _renew_subs 2024-03-29T10:36:07.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.788+0000 7f1c8d94d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:36:07.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.788+0000 7f1c8d94d640 1 -- 172.21.15.84:0/2587763120 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c881db5e0 con 0x7f1c88149c50 2024-03-29T10:36:07.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.788+0000 7f1c8d94d640 1 librados: init done 2024-03-29T10:36:07.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.788+0000 7f1c6ffff640 1 -- 172.21.15.84:0/2587763120 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c78004b00 con 0x7f1c88149c50 2024-03-29T10:36:07.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.788+0000 7f1c6ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:07.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.788+0000 7f1c84ff9640 4 set_mon_vals no callback set 2024-03-29T10:36:07.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.789+0000 7f1c6ffff640 1 -- 172.21.15.84:0/2587763120 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f1c78004f10 con 0x7f1c88149c50 2024-03-29T10:36:07.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.789+0000 7f1c6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:07.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.789+0000 7f1c6ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:07.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.789+0000 7f1c6ffff640 10 monclient: dump: 2024-03-29T10:36:07.790 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:07.790 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:07.790 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:07.790 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:07.790 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:07.790 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:07.790 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:07.790 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:07.790 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:07.790 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:07.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.790+0000 7f1c6dffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:07.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.790+0000 7f1c6dffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:07.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.790+0000 7f1c6dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:36:07.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.790+0000 7f1c6dffb640 1 -- 172.21.15.84:0/2587763120 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c88075f10 con 0x7f1c88149c50 2024-03-29T10:36:07.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.791+0000 7f1c6ffff640 1 -- 172.21.15.84:0/2587763120 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f1c78005070 con 0x7f1c88149c50 2024-03-29T10:36:07.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.792+0000 7f1c6ffff640 1 -- 172.21.15.84:0/2587763120 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f1c78084b60 con 0x7f1c88149c50 2024-03-29T10:36:07.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.792+0000 7f1c6ffff640 10 client.4273.objecter ms_dispatch 0x7f1c88000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:36:07.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.792+0000 7f1c6ffff640 3 client.4273.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:36:07.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.792+0000 7f1c6ffff640 3 client.4273.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:36:07.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.792+0000 7f1c86ffd640 10 monclient: get_auth_request con 0x7f1c7406e000 auth_method 0 2024-03-29T10:36:07.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.792+0000 7f1c6ffff640 20 client.4273.objecter dump_active .. 0 homeless 2024-03-29T10:36:07.791 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.793+0000 7f1c6ffff640 10 client.4273.objecter ms_handle_connect 0x7f1c7406e000 2024-03-29T10:36:07.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.798+0000 7f1c6ffff640 1 -- 172.21.15.84:0/2587763120 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1c780504a0 con 0x7f1c88149c50 2024-03-29T10:36:07.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.799+0000 7f1c6ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:07.792 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:07.799+0000 7f1c6ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:36:08.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:08.190+0000 7f1c6dffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:08.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:08.190+0000 7f1c6dffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:08.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:08.190+0000 7f1c6dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:36:08.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:08.190+0000 7f1c6dffb640 1 -- 172.21.15.84:0/2587763120 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c881defd0 con 0x7f1c88149c50 2024-03-29T10:36:08.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:08.191+0000 7f1c6ffff640 1 -- 172.21.15.84:0/2587763120 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f1c78084210 con 0x7f1c88149c50 2024-03-29T10:36:08.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:08.191+0000 7f1c6ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:08.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:08.191+0000 7f1c6ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:36:08.177 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:36:08.177 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:36:08.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:08.195+0000 7f1c6dffb640 10 librados: watch_flush enter 2024-03-29T10:36:08.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:08.195+0000 7f1c6dffb640 10 librados: watch_flush exit 2024-03-29T10:36:08.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:08.195+0000 7f1c6dffb640 20 client.4273.objecter shutdown clearing up homeless session... 2024-03-29T10:36:08.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:08.195+0000 7f1c6dffb640 10 client.4273.objecter successfully canceled tick 2024-03-29T10:36:08.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:08.195+0000 7f1c6dffb640 1 -- 172.21.15.84:0/2587763120 >> v1:172.21.15.86:6816/3337225337 conn(0x7f1c7406e000 legacy=0x7f1c740704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:08.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:08.195+0000 7f1c6dffb640 10 monclient: shutdown 2024-03-29T10:36:08.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:08.195+0000 7f1c6dffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:08.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:08.195+0000 7f1c6dffb640 1 -- 172.21.15.84:0/2587763120 >> v1:172.21.15.86:6789/0 conn(0x7f1c88149c50 legacy=0x7f1c8813d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:08.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:08.195+0000 7f1c6dffb640 1 -- 172.21.15.84:0/2587763120 shutdown_connections 2024-03-29T10:36:08.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:08.195+0000 7f1c6dffb640 1 -- 172.21.15.84:0/2587763120 >> 172.21.15.84:0/2587763120 conn(0x7f1c880b3c50 msgr2=0x7f1c880b4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:08.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:08.196+0000 7f1c6dffb640 1 -- 172.21.15.84:0/2587763120 shutdown_connections 2024-03-29T10:36:08.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:08.196+0000 7f1c6dffb640 1 -- 172.21.15.84:0/2587763120 wait complete. 2024-03-29T10:36:08.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:08.196+0000 7f1c6dffb640 1 librados: shutdown 2024-03-29T10:36:08.195 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:36:15.002 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:36:15.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.144+0000 7fa9ef1ba640 10 monclient: get_monmap_and_config 2024-03-29T10:36:15.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.144+0000 7fa9ef1ba640 10 monclient: build_initial_monmap 2024-03-29T10:36:15.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.145+0000 7fa9ef1ba640 10 monclient: monmap: 2024-03-29T10:36:15.130 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:36:15.130 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:15.130 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:36:15.145950+0000 2024-03-29T10:36:15.130 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:36:15.145950+0000 2024-03-29T10:36:15.130 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:15.130 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:15.130 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:36:15.130 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:36:15.130 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:36:15.130 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:15.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.146+0000 7fa9ef1ba640 1 Processor -- start 2024-03-29T10:36:15.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.146+0000 7fa9ef1ba640 1 -- start start 2024-03-29T10:36:15.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.146+0000 7fa9ef1ba640 10 monclient: init 2024-03-29T10:36:15.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.146+0000 7fa9ef1ba640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:15.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.146+0000 7fa9ef1ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:36:15.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.146+0000 7fa9ef1ba640 10 monclient(hunting): picked mon.noname-a con 0x7fa9e81457f0 addr v1:172.21.15.84:6789/0 2024-03-29T10:36:15.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.147+0000 7fa9ef1ba640 10 monclient(hunting): picked mon.noname-b con 0x7fa9e813f650 addr v1:172.21.15.84:6790/0 2024-03-29T10:36:15.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.147+0000 7fa9e7fff640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fa9e813f650 0x7fa9e80b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:40984/0 (socket says 172.21.15.84:40984) 2024-03-29T10:36:15.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.147+0000 7fa9e7fff640 1 -- 172.21.15.84:0/556035486 learned_addr learned my addr 172.21.15.84:0/556035486 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:36:15.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.147+0000 7fa9ef1ba640 10 monclient(hunting): picked mon.noname-c con 0x7fa9e80a9d70 addr v1:172.21.15.86:6789/0 2024-03-29T10:36:15.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.147+0000 7fa9ef1ba640 1 -- 172.21.15.84:0/556035486 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e80aaef0 con 0x7fa9e81457f0 2024-03-29T10:36:15.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.147+0000 7fa9ef1ba640 1 -- 172.21.15.84:0/556035486 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e8145bd0 con 0x7fa9e80a9d70 2024-03-29T10:36:15.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.147+0000 7fa9ef1ba640 1 -- 172.21.15.84:0/556035486 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e8145eb0 con 0x7fa9e813f650 2024-03-29T10:36:15.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.147+0000 7fa9ef1ba640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:15.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.147+0000 7fa9ef1ba640 10 monclient(hunting): authenticate will time out at 877.410423s 2024-03-29T10:36:15.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.147+0000 7fa9e77fe640 1 -- 172.21.15.84:0/556035486 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2442163869 0 0) 0x7fa9e8145eb0 con 0x7fa9e813f650 2024-03-29T10:36:15.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.147+0000 7fa9e77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:15.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.147+0000 7fa9e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:15.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.147+0000 7fa9e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:15.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.147+0000 7fa9e77fe640 10 monclient(hunting): my global_id is 4286 2024-03-29T10:36:15.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.147+0000 7fa9e77fe640 1 -- 172.21.15.84:0/556035486 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9d0003520 con 0x7fa9e813f650 2024-03-29T10:36:15.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.147+0000 7fa9e77fe640 1 -- 172.21.15.84:0/556035486 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1622965278 0 0) 0x7fa9e8145bd0 con 0x7fa9e80a9d70 2024-03-29T10:36:15.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.147+0000 7fa9e77fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:15.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.147+0000 7fa9e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:15.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.147+0000 7fa9e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:15.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.147+0000 7fa9e77fe640 10 monclient(hunting): my global_id is 4276 2024-03-29T10:36:15.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.147+0000 7fa9e77fe640 1 -- 172.21.15.84:0/556035486 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9e8145eb0 con 0x7fa9e80a9d70 2024-03-29T10:36:15.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.148+0000 7fa9e77fe640 1 -- 172.21.15.84:0/556035486 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 837425852 0 0) 0x7fa9d0003520 con 0x7fa9e813f650 2024-03-29T10:36:15.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.148+0000 7fa9e77fe640 1 -- 172.21.15.84:0/556035486 >> v1:172.21.15.86:6789/0 conn(0x7fa9e80a9d70 legacy=0x7fa9e814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:15.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.148+0000 7fa9e77fe640 1 -- 172.21.15.84:0/556035486 >> v1:172.21.15.84:6789/0 conn(0x7fa9e81457f0 legacy=0x7fa9e813ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:36:15.137 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.149+0000 7fa9e77fe640 10 monclient: _finish_hunting 0 2024-03-29T10:36:15.137 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.149+0000 7fa9e77fe640 1 monclient: found mon.noname-b 2024-03-29T10:36:15.137 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.149+0000 7fa9e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:15.137 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.149+0000 7fa9e77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:36:15.137 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.149+0000 7fa9e77fe640 1 -- 172.21.15.84:0/556035486 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e8146340 con 0x7fa9e813f650 2024-03-29T10:36:15.137 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.149+0000 7fa9e77fe640 10 monclient: _finish_auth 0 2024-03-29T10:36:15.137 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.149+0000 7fa9e77fe640 10 monclient: _check_auth_tickets 2024-03-29T10:36:15.137 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.149+0000 7fa9e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:15.137 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.149+0000 7fa9e77fe640 1 -- 172.21.15.84:0/556035486 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa9dc001690 con 0x7fa9e813f650 2024-03-29T10:36:15.137 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.149+0000 7fa9e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:15.137 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.149+0000 7fa9e77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:36:15.137 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.149+0000 7fa9e77fe640 10 monclient: dump: 2024-03-29T10:36:15.137 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:15.138 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:15.138 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:15.138 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:15.138 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:15.138 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:15.138 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:15.138 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:15.138 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:15.138 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:15.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.149+0000 7fa9ef1ba640 5 monclient: authenticate success, global_id 4286 2024-03-29T10:36:15.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.149+0000 7fa9ef1ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:15.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.150+0000 7fa9e77fe640 1 -- 172.21.15.84:0/556035486 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9dc0018b0 con 0x7fa9e813f650 2024-03-29T10:36:15.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.150+0000 7fa9e77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:15.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.150+0000 7fa9e77fe640 1 -- 172.21.15.84:0/556035486 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa9dc001c40 con 0x7fa9e813f650 2024-03-29T10:36:15.139 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.150+0000 7fa9ef1ba640 10 monclient: get_monmap_and_config success 2024-03-29T10:36:15.139 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.150+0000 7fa9ef1ba640 4 set_mon_vals no callback set 2024-03-29T10:36:15.139 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.150+0000 7fa9e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:15.139 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.150+0000 7fa9e77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:15.139 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.150+0000 7fa9e77fe640 10 monclient: dump: 2024-03-29T10:36:15.139 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:15.139 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:15.139 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:15.139 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:15.139 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:15.139 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:15.139 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:15.139 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:15.139 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:15.139 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:15.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.151+0000 7fa9ef1ba640 10 monclient: shutdown 2024-03-29T10:36:15.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.151+0000 7fa9ef1ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:15.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.151+0000 7fa9ef1ba640 1 -- 172.21.15.84:0/556035486 >> v1:172.21.15.84:6790/0 conn(0x7fa9e813f650 legacy=0x7fa9e80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:15.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.151+0000 7fa9ef1ba640 1 -- 172.21.15.84:0/556035486 shutdown_connections 2024-03-29T10:36:15.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.151+0000 7fa9ef1ba640 1 -- 172.21.15.84:0/556035486 >> 172.21.15.84:0/556035486 conn(0x7fa9e8066a00 msgr2=0x7fa9e8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:15.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.152+0000 7fa9ef1ba640 1 -- 172.21.15.84:0/556035486 shutdown_connections 2024-03-29T10:36:15.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.152+0000 7fa9ef1ba640 1 -- 172.21.15.84:0/556035486 wait complete. 2024-03-29T10:36:15.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.153+0000 7fa9ef1ba640 10 monclient: build_initial_monmap 2024-03-29T10:36:15.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.153+0000 7fa9ef1ba640 10 monclient: monmap: 2024-03-29T10:36:15.142 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:36:15.142 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:36:15.142 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:36:15.142 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:36:15.143 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:15.143 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:15.143 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:36:15.143 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:36:15.143 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:36:15.143 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:15.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.154+0000 7fa9ef1ba640 1 librados: starting msgr at 2024-03-29T10:36:15.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.154+0000 7fa9ef1ba640 1 librados: starting objecter 2024-03-29T10:36:15.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.154+0000 7fa9ef1ba640 1 Processor -- start 2024-03-29T10:36:15.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.154+0000 7fa9ef1ba640 1 -- start start 2024-03-29T10:36:15.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.154+0000 7fa9ef1ba640 1 librados: setting wanted keys 2024-03-29T10:36:15.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.154+0000 7fa9ef1ba640 1 librados: calling monclient init 2024-03-29T10:36:15.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.154+0000 7fa9ef1ba640 10 monclient: init 2024-03-29T10:36:15.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.155+0000 7fa9ef1ba640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:15.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.155+0000 7fa9ef1ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:36:15.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.155+0000 7fa9ef1ba640 10 monclient(hunting): picked mon.noname-b con 0x7fa9e80a9d70 addr v1:172.21.15.86:6789/0 2024-03-29T10:36:15.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.155+0000 7fa9ef1ba640 10 monclient(hunting): picked mon.noname-c con 0x7fa9e813f650 addr v1:172.21.15.84:6790/0 2024-03-29T10:36:15.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.155+0000 7fa9ef1ba640 10 monclient(hunting): picked mon.noname-a con 0x7fa9e81457f0 addr v1:172.21.15.84:6789/0 2024-03-29T10:36:15.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.155+0000 7fa9ef1ba640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e81dba60 con 0x7fa9e81457f0 2024-03-29T10:36:15.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.155+0000 7fa9ef1ba640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e8137f20 con 0x7fa9e80a9d70 2024-03-29T10:36:15.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.155+0000 7fa9ef1ba640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e81dc600 con 0x7fa9e813f650 2024-03-29T10:36:15.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.155+0000 7fa9ef1ba640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:15.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.155+0000 7fa9ef1ba640 10 monclient(hunting): authenticate will time out at 877.418865s 2024-03-29T10:36:15.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.155+0000 7fa9ecf56640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa9e80a9d70 0x7fa9e81dbef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:42442/0 (socket says 172.21.15.84:42442) 2024-03-29T10:36:15.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.155+0000 7fa9ecf56640 1 -- 172.21.15.84:0/85236305 learned_addr learned my addr 172.21.15.84:0/85236305 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:36:15.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.155+0000 7fa9e57fa640 10 client.?.objecter ms_handle_connect 0x7fa9e80a9d70 2024-03-29T10:36:15.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.155+0000 7fa9e57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:15.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.156+0000 7fa9e57fa640 10 client.?.objecter ms_handle_connect 0x7fa9e813f650 2024-03-29T10:36:15.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.156+0000 7fa9e57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:15.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.157+0000 7fa9e57fa640 1 -- 172.21.15.84:0/85236305 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3584318226 0 0) 0x7fa9e8137f20 con 0x7fa9e80a9d70 2024-03-29T10:36:15.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.157+0000 7fa9e57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:36:15.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.157+0000 7fa9e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:15.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.157+0000 7fa9e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:15.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.157+0000 7fa9e57fa640 10 monclient(hunting): my global_id is 4279 2024-03-29T10:36:15.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.157+0000 7fa9e57fa640 1 -- 172.21.15.84:0/85236305 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9d4004eb0 con 0x7fa9e80a9d70 2024-03-29T10:36:15.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.157+0000 7fa9e57fa640 1 -- 172.21.15.84:0/85236305 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2276822890 0 0) 0x7fa9e81dc600 con 0x7fa9e813f650 2024-03-29T10:36:15.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.157+0000 7fa9e57fa640 10 monclient(hunting): _negotiate 2024-03-29T10:36:15.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.157+0000 7fa9e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:15.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.157+0000 7fa9e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:15.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.157+0000 7fa9e57fa640 10 monclient(hunting): my global_id is 4289 2024-03-29T10:36:15.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.157+0000 7fa9e57fa640 1 -- 172.21.15.84:0/85236305 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9e8137f20 con 0x7fa9e813f650 2024-03-29T10:36:15.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.157+0000 7fa9e57fa640 1 -- 172.21.15.84:0/85236305 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3876768799 0 0) 0x7fa9d4004eb0 con 0x7fa9e80a9d70 2024-03-29T10:36:15.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.157+0000 7fa9e57fa640 1 -- 172.21.15.84:0/85236305 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9e81dc600 con 0x7fa9e80a9d70 2024-03-29T10:36:15.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.157+0000 7fa9e57fa640 1 -- 172.21.15.84:0/85236305 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa9e0001b50 con 0x7fa9e80a9d70 2024-03-29T10:36:15.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.157+0000 7fa9e57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:15.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.157+0000 7fa9e57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:36:15.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.157+0000 7fa9e57fa640 10 monclient(hunting): dump: 2024-03-29T10:36:15.146 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:15.146 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:15.146 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:15.146 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:15.146 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:15.146 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:15.146 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:15.146 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:15.146 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:15.147 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:15.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.157+0000 7fa9e57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:36:15.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.158+0000 7fa9e57fa640 1 -- 172.21.15.84:0/85236305 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2792953998 0 0) 0x7fa9e81dc600 con 0x7fa9e80a9d70 2024-03-29T10:36:15.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.158+0000 7fa9e57fa640 1 -- 172.21.15.84:0/85236305 >> v1:172.21.15.84:6790/0 conn(0x7fa9e813f650 legacy=0x7fa9e8137100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:15.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.158+0000 7fa9e57fa640 1 -- 172.21.15.84:0/85236305 >> v1:172.21.15.84:6789/0 conn(0x7fa9e81457f0 legacy=0x7fa9e8137810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:36:15.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.159+0000 7fa9e57fa640 10 monclient: _finish_hunting 0 2024-03-29T10:36:15.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.159+0000 7fa9e57fa640 1 monclient: found mon.b 2024-03-29T10:36:15.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.159+0000 7fa9e57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:15.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.159+0000 7fa9e57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:36:15.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.159+0000 7fa9e57fa640 1 -- 172.21.15.84:0/85236305 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e8142bb0 con 0x7fa9e80a9d70 2024-03-29T10:36:15.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.159+0000 7fa9e57fa640 10 monclient: _finish_auth 0 2024-03-29T10:36:15.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.159+0000 7fa9e57fa640 10 monclient: _check_auth_tickets 2024-03-29T10:36:15.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.159+0000 7fa9e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:15.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.159+0000 7fa9ef1ba640 5 monclient: authenticate success, global_id 4279 2024-03-29T10:36:15.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.159+0000 7fa9ef1ba640 10 monclient: _renew_subs 2024-03-29T10:36:15.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.159+0000 7fa9ef1ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:36:15.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.159+0000 7fa9ef1ba640 1 -- 172.21.15.84:0/85236305 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e8142e00 con 0x7fa9e80a9d70 2024-03-29T10:36:15.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.159+0000 7fa9ef1ba640 10 client.4279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:36:15.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.159+0000 7fa9ef1ba640 10 monclient: _renew_subs 2024-03-29T10:36:15.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.159+0000 7fa9ef1ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:36:15.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.159+0000 7fa9ef1ba640 1 -- 172.21.15.84:0/85236305 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e81422f0 con 0x7fa9e80a9d70 2024-03-29T10:36:15.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.159+0000 7fa9ef1ba640 1 librados: init done 2024-03-29T10:36:15.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.160+0000 7fa9e57fa640 1 -- 172.21.15.84:0/85236305 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9e0002060 con 0x7fa9e80a9d70 2024-03-29T10:36:15.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.160+0000 7fa9e57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:15.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.160+0000 7fa9e57fa640 1 -- 172.21.15.84:0/85236305 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa9e0004d80 con 0x7fa9e80a9d70 2024-03-29T10:36:15.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.160+0000 7fa9e57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:15.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.160+0000 7fa9e57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:15.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.160+0000 7fa9e57fa640 10 monclient: dump: 2024-03-29T10:36:15.152 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:15.152 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:15.152 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:15.152 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:15.152 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:15.152 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:15.152 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:15.152 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:15.152 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:15.152 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:15.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.160+0000 7fa9e6ffd640 4 set_mon_vals no callback set 2024-03-29T10:36:15.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.160+0000 7fa9ef1ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:15.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.161+0000 7fa9e57fa640 1 -- 172.21.15.84:0/85236305 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7fa9e001ad50 con 0x7fa9e80a9d70 2024-03-29T10:36:15.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.161+0000 7fa9ef1ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:15.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.161+0000 7fa9ef1ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:36:15.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.161+0000 7fa9ef1ba640 1 -- 172.21.15.84:0/85236305 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e8075f10 con 0x7fa9e80a9d70 2024-03-29T10:36:15.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.162+0000 7fa9e57fa640 1 -- 172.21.15.84:0/85236305 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7fa9e0084cf0 con 0x7fa9e80a9d70 2024-03-29T10:36:15.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.162+0000 7fa9e57fa640 10 client.4279.objecter ms_dispatch 0x7fa9e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:36:15.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.162+0000 7fa9e57fa640 3 client.4279.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:36:15.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.162+0000 7fa9e57fa640 3 client.4279.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:36:15.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.162+0000 7fa9e57fa640 20 client.4279.objecter dump_active .. 0 homeless 2024-03-29T10:36:15.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.164+0000 7fa9e7fff640 10 monclient: get_auth_request con 0x7fa9d406dd20 auth_method 0 2024-03-29T10:36:15.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.165+0000 7fa9e57fa640 10 client.4279.objecter ms_handle_connect 0x7fa9d406dd20 2024-03-29T10:36:15.154 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.169+0000 7fa9e57fa640 1 -- 172.21.15.84:0/85236305 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa9e0050630 con 0x7fa9e80a9d70 2024-03-29T10:36:15.154 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.169+0000 7fa9e57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:15.154 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.169+0000 7fa9e57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:36:15.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.467+0000 7fa9ef1ba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:15.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.468+0000 7fa9ef1ba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:15.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.468+0000 7fa9ef1ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:36:15.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.468+0000 7fa9ef1ba640 1 -- 172.21.15.84:0/85236305 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9e80b3680 con 0x7fa9e80a9d70 2024-03-29T10:36:15.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.468+0000 7fa9e57fa640 1 -- 172.21.15.84:0/85236305 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7fa9e00843a0 con 0x7fa9e80a9d70 2024-03-29T10:36:15.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.468+0000 7fa9e57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:15.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.468+0000 7fa9e57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:36:15.454 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:36:15.454 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:36:15.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.471+0000 7fa9ef1ba640 10 librados: watch_flush enter 2024-03-29T10:36:15.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.471+0000 7fa9ef1ba640 10 librados: watch_flush exit 2024-03-29T10:36:15.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.471+0000 7fa9ef1ba640 20 client.4279.objecter shutdown clearing up homeless session... 2024-03-29T10:36:15.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.471+0000 7fa9ef1ba640 10 client.4279.objecter successfully canceled tick 2024-03-29T10:36:15.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.472+0000 7fa9ef1ba640 1 -- 172.21.15.84:0/85236305 >> v1:172.21.15.86:6816/3337225337 conn(0x7fa9d406dd20 legacy=0x7fa9d40701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:15.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.472+0000 7fa9ef1ba640 10 monclient: shutdown 2024-03-29T10:36:15.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.472+0000 7fa9ef1ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:15.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.472+0000 7fa9ef1ba640 1 -- 172.21.15.84:0/85236305 >> v1:172.21.15.86:6789/0 conn(0x7fa9e80a9d70 legacy=0x7fa9e81dbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:15.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.472+0000 7fa9ef1ba640 1 -- 172.21.15.84:0/85236305 shutdown_connections 2024-03-29T10:36:15.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.472+0000 7fa9ef1ba640 1 -- 172.21.15.84:0/85236305 >> 172.21.15.84:0/85236305 conn(0x7fa9e8066a00 msgr2=0x7fa9e80b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:15.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.472+0000 7fa9ef1ba640 1 -- 172.21.15.84:0/85236305 shutdown_connections 2024-03-29T10:36:15.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.472+0000 7fa9ef1ba640 1 -- 172.21.15.84:0/85236305 wait complete. 2024-03-29T10:36:15.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:15.472+0000 7fa9ef1ba640 1 librados: shutdown 2024-03-29T10:36:15.470 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:36:22.277 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:36:22.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.427+0000 7f1667e88640 10 monclient: get_monmap_and_config 2024-03-29T10:36:22.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.427+0000 7f1667e88640 10 monclient: build_initial_monmap 2024-03-29T10:36:22.412 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.427+0000 7f1667e88640 10 monclient: monmap: 2024-03-29T10:36:22.412 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:36:22.412 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:22.412 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:36:22.428239+0000 2024-03-29T10:36:22.412 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:36:22.428239+0000 2024-03-29T10:36:22.412 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:22.413 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:22.413 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:36:22.413 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:36:22.413 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:36:22.413 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:22.415 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.428+0000 7f1667e88640 1 Processor -- start 2024-03-29T10:36:22.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.428+0000 7f1667e88640 1 -- start start 2024-03-29T10:36:22.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.428+0000 7f1667e88640 10 monclient: init 2024-03-29T10:36:22.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.429+0000 7f1667e88640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:22.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.429+0000 7f1667e88640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:36:22.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.429+0000 7f1667e88640 10 monclient(hunting): picked mon.noname-c con 0x7f16601457f0 addr v1:172.21.15.86:6789/0 2024-03-29T10:36:22.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.429+0000 7f1667e88640 10 monclient(hunting): picked mon.noname-a con 0x7f166013f650 addr v1:172.21.15.84:6789/0 2024-03-29T10:36:22.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.429+0000 7f1667e88640 10 monclient(hunting): picked mon.noname-b con 0x7f16600a9d70 addr v1:172.21.15.84:6790/0 2024-03-29T10:36:22.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.429+0000 7f1667e88640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16600aaef0 con 0x7f166013f650 2024-03-29T10:36:22.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.429+0000 7f1667e88640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1660145bd0 con 0x7f16601457f0 2024-03-29T10:36:22.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.429+0000 7f1667e88640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1660145eb0 con 0x7f16600a9d70 2024-03-29T10:36:22.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.429+0000 7f1667e88640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:22.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.429+0000 7f1667e88640 10 monclient(hunting): authenticate will time out at 884.692333s 2024-03-29T10:36:22.416 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.429+0000 7f1666425640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f16600a9d70 0x7f166014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:53932/0 (socket says 172.21.15.84:53932) 2024-03-29T10:36:22.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.429+0000 7f1666425640 1 -- 172.21.15.84:0/2762178480 learned_addr learned my addr 172.21.15.84:0/2762178480 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:36:22.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.429+0000 7f1664c22640 1 -- 172.21.15.84:0/2762178480 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1270576090 0 0) 0x7f1660145eb0 con 0x7f16600a9d70 2024-03-29T10:36:22.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.429+0000 7f1664c22640 10 monclient(hunting): _negotiate 2024-03-29T10:36:22.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.429+0000 7f1664c22640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:22.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.429+0000 7f1664c22640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:22.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.429+0000 7f1664c22640 10 monclient(hunting): my global_id is 4292 2024-03-29T10:36:22.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.429+0000 7f1664c22640 1 -- 172.21.15.84:0/2762178480 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f164c003520 con 0x7f16600a9d70 2024-03-29T10:36:22.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.430+0000 7f1664c22640 1 -- 172.21.15.84:0/2762178480 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1624133181 0 0) 0x7f164c003520 con 0x7f16600a9d70 2024-03-29T10:36:22.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.430+0000 7f1664c22640 1 -- 172.21.15.84:0/2762178480 >> v1:172.21.15.86:6789/0 conn(0x7f16601457f0 legacy=0x7f166013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:36:22.417 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.430+0000 7f1664c22640 1 -- 172.21.15.84:0/2762178480 >> v1:172.21.15.84:6789/0 conn(0x7f166013f650 legacy=0x7f16600b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:36:22.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.433+0000 7f1664c22640 10 monclient: _finish_hunting 0 2024-03-29T10:36:22.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.433+0000 7f1664c22640 1 monclient: found mon.noname-b 2024-03-29T10:36:22.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.433+0000 7f1664c22640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:22.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.433+0000 7f1664c22640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:36:22.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.433+0000 7f1664c22640 1 -- 172.21.15.84:0/2762178480 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1660146340 con 0x7f16600a9d70 2024-03-29T10:36:22.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.433+0000 7f1664c22640 10 monclient: _finish_auth 0 2024-03-29T10:36:22.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.433+0000 7f1664c22640 10 monclient: _check_auth_tickets 2024-03-29T10:36:22.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.433+0000 7f1664c22640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:22.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.433+0000 7f1664c22640 1 -- 172.21.15.84:0/2762178480 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f1658001a10 con 0x7f16600a9d70 2024-03-29T10:36:22.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.433+0000 7f1664c22640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:22.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.433+0000 7f1664c22640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:36:22.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.433+0000 7f1664c22640 10 monclient: dump: 2024-03-29T10:36:22.423 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:22.423 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:22.423 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:22.423 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:22.423 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:22.423 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:22.423 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:22.424 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:22.424 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:22.424 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:22.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.433+0000 7f1667e88640 5 monclient: authenticate success, global_id 4292 2024-03-29T10:36:22.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.433+0000 7f1667e88640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:22.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.433+0000 7f1664c22640 1 -- 172.21.15.84:0/2762178480 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1658001be0 con 0x7f16600a9d70 2024-03-29T10:36:22.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.433+0000 7f1664c22640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:22.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.433+0000 7f1664c22640 1 -- 172.21.15.84:0/2762178480 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f1658002030 con 0x7f16600a9d70 2024-03-29T10:36:22.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.433+0000 7f1664c22640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:22.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.433+0000 7f1664c22640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:22.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.433+0000 7f1664c22640 10 monclient: dump: 2024-03-29T10:36:22.425 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:22.425 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:22.425 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:22.425 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:22.425 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:22.425 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:22.425 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:22.425 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:22.425 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:22.425 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:22.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.433+0000 7f1667e88640 10 monclient: get_monmap_and_config success 2024-03-29T10:36:22.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.433+0000 7f1667e88640 4 set_mon_vals no callback set 2024-03-29T10:36:22.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.434+0000 7f1667e88640 10 monclient: shutdown 2024-03-29T10:36:22.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.434+0000 7f1667e88640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:22.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.434+0000 7f1667e88640 1 -- 172.21.15.84:0/2762178480 >> v1:172.21.15.84:6790/0 conn(0x7f16600a9d70 legacy=0x7f166014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:22.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.434+0000 7f1667e88640 1 -- 172.21.15.84:0/2762178480 shutdown_connections 2024-03-29T10:36:22.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.434+0000 7f1667e88640 1 -- 172.21.15.84:0/2762178480 >> 172.21.15.84:0/2762178480 conn(0x7f1660066a00 msgr2=0x7f1660066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:22.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.434+0000 7f1667e88640 1 -- 172.21.15.84:0/2762178480 shutdown_connections 2024-03-29T10:36:22.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.435+0000 7f1667e88640 1 -- 172.21.15.84:0/2762178480 wait complete. 2024-03-29T10:36:22.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.436+0000 7f1667e88640 10 monclient: build_initial_monmap 2024-03-29T10:36:22.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.436+0000 7f1667e88640 10 monclient: monmap: 2024-03-29T10:36:22.427 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:36:22.427 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:36:22.427 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:36:22.427 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:36:22.427 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:22.427 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:22.427 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:36:22.427 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:36:22.427 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:36:22.428 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:22.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.436+0000 7f1667e88640 1 librados: starting msgr at 2024-03-29T10:36:22.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.436+0000 7f1667e88640 1 librados: starting objecter 2024-03-29T10:36:22.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.437+0000 7f1667e88640 1 Processor -- start 2024-03-29T10:36:22.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.437+0000 7f1667e88640 1 -- start start 2024-03-29T10:36:22.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.437+0000 7f1667e88640 1 librados: setting wanted keys 2024-03-29T10:36:22.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.437+0000 7f1667e88640 1 librados: calling monclient init 2024-03-29T10:36:22.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.437+0000 7f1667e88640 10 monclient: init 2024-03-29T10:36:22.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.438+0000 7f1667e88640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:22.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.438+0000 7f1667e88640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:36:22.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.438+0000 7f1667e88640 10 monclient(hunting): picked mon.noname-a con 0x7f16600a9d70 addr v1:172.21.15.84:6789/0 2024-03-29T10:36:22.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.438+0000 7f1667e88640 10 monclient(hunting): picked mon.noname-c con 0x7f166013f650 addr v1:172.21.15.84:6790/0 2024-03-29T10:36:22.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.439+0000 7f1665423640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f166013f650 0x7f16601dedd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:55352/0 (socket says 172.21.15.84:55352) 2024-03-29T10:36:22.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.439+0000 7f1665423640 1 -- 172.21.15.84:0/3880406771 learned_addr learned my addr 172.21.15.84:0/3880406771 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:36:22.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.439+0000 7f1667e88640 10 monclient(hunting): picked mon.noname-b con 0x7f16601457f0 addr v1:172.21.15.86:6789/0 2024-03-29T10:36:22.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.439+0000 7f1667e88640 1 -- 172.21.15.84:0/3880406771 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f166013e3a0 con 0x7f16600a9d70 2024-03-29T10:36:22.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.439+0000 7f16567fc640 10 client.?.objecter ms_handle_connect 0x7f166013f650 2024-03-29T10:36:22.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.439+0000 7f16567fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:22.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.439+0000 7f16567fc640 10 client.?.objecter ms_handle_connect 0x7f16601457f0 2024-03-29T10:36:22.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.439+0000 7f16567fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:22.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.439+0000 7f1667e88640 1 -- 172.21.15.84:0/3880406771 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f166013e120 con 0x7f16601457f0 2024-03-29T10:36:22.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.439+0000 7f1667e88640 1 -- 172.21.15.84:0/3880406771 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f166013cd20 con 0x7f166013f650 2024-03-29T10:36:22.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.440+0000 7f1667e88640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:22.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.440+0000 7f1667e88640 10 monclient(hunting): authenticate will time out at 884.703179s 2024-03-29T10:36:22.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.440+0000 7f16567fc640 1 -- 172.21.15.84:0/3880406771 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582518850 0 0) 0x7f166013e120 con 0x7f16601457f0 2024-03-29T10:36:22.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.440+0000 7f16567fc640 10 monclient(hunting): _negotiate 2024-03-29T10:36:22.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.440+0000 7f16567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:22.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.440+0000 7f16567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:22.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.440+0000 7f16567fc640 10 monclient(hunting): my global_id is 4282 2024-03-29T10:36:22.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.440+0000 7f16567fc640 1 -- 172.21.15.84:0/3880406771 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1644004eb0 con 0x7f16601457f0 2024-03-29T10:36:22.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.440+0000 7f16567fc640 1 -- 172.21.15.84:0/3880406771 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610531278 0 0) 0x7f166013cd20 con 0x7f166013f650 2024-03-29T10:36:22.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.440+0000 7f16567fc640 10 monclient(hunting): _negotiate 2024-03-29T10:36:22.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.440+0000 7f16567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:22.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.440+0000 7f16567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:22.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.440+0000 7f16567fc640 10 monclient(hunting): my global_id is 4295 2024-03-29T10:36:22.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.440+0000 7f16567fc640 1 -- 172.21.15.84:0/3880406771 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f166013e120 con 0x7f166013f650 2024-03-29T10:36:22.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.440+0000 7f16567fc640 1 -- 172.21.15.84:0/3880406771 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2653396646 0 0) 0x7f1644004eb0 con 0x7f16601457f0 2024-03-29T10:36:22.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.440+0000 7f16567fc640 1 -- 172.21.15.84:0/3880406771 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f166013cd20 con 0x7f16601457f0 2024-03-29T10:36:22.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.440+0000 7f16567fc640 1 -- 172.21.15.84:0/3880406771 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f1658001ef0 con 0x7f16601457f0 2024-03-29T10:36:22.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.440+0000 7f16567fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:22.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.440+0000 7f16567fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:36:22.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.440+0000 7f16567fc640 10 monclient(hunting): dump: 2024-03-29T10:36:22.438 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:22.438 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:22.438 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:22.438 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:22.438 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:22.438 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:22.438 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:22.438 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:22.438 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:22.438 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:22.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.440+0000 7f16567fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:36:22.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.441+0000 7f16567fc640 1 -- 172.21.15.84:0/3880406771 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2958606537 0 0) 0x7f166013e120 con 0x7f166013f650 2024-03-29T10:36:22.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.441+0000 7f16567fc640 1 -- 172.21.15.84:0/3880406771 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1644004eb0 con 0x7f166013f650 2024-03-29T10:36:22.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.441+0000 7f16567fc640 1 -- 172.21.15.84:0/3880406771 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1992267696 0 0) 0x7f166013cd20 con 0x7f16601457f0 2024-03-29T10:36:22.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.441+0000 7f16567fc640 1 -- 172.21.15.84:0/3880406771 >> v1:172.21.15.84:6790/0 conn(0x7f166013f650 legacy=0x7f16601dedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:22.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.441+0000 7f16567fc640 1 -- 172.21.15.84:0/3880406771 >> v1:172.21.15.84:6789/0 conn(0x7f16600a9d70 legacy=0x7f166013c610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:36:22.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.441+0000 7f16567fc640 10 monclient: _finish_hunting 0 2024-03-29T10:36:22.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.441+0000 7f16567fc640 1 monclient: found mon.b 2024-03-29T10:36:22.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.441+0000 7f16567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:22.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.441+0000 7f16567fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:36:22.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.441+0000 7f16567fc640 1 -- 172.21.15.84:0/3880406771 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1660141450 con 0x7f16601457f0 2024-03-29T10:36:22.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.441+0000 7f16567fc640 10 monclient: _finish_auth 0 2024-03-29T10:36:22.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.441+0000 7f16567fc640 10 monclient: _check_auth_tickets 2024-03-29T10:36:22.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.441+0000 7f16567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:22.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.441+0000 7f1667e88640 5 monclient: authenticate success, global_id 4282 2024-03-29T10:36:22.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.441+0000 7f1667e88640 10 monclient: _renew_subs 2024-03-29T10:36:22.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.441+0000 7f1667e88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:36:22.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.441+0000 7f1667e88640 1 -- 172.21.15.84:0/3880406771 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1660141730 con 0x7f16601457f0 2024-03-29T10:36:22.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.442+0000 7f1667e88640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:36:22.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.442+0000 7f1667e88640 10 monclient: _renew_subs 2024-03-29T10:36:22.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.442+0000 7f1667e88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:36:22.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.442+0000 7f1667e88640 1 -- 172.21.15.84:0/3880406771 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1660142b60 con 0x7f16601457f0 2024-03-29T10:36:22.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.442+0000 7f1667e88640 1 librados: init done 2024-03-29T10:36:22.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.442+0000 7f16567fc640 1 -- 172.21.15.84:0/3880406771 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1658001390 con 0x7f16601457f0 2024-03-29T10:36:22.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.442+0000 7f16567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:22.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.442+0000 7f16567fc640 1 -- 172.21.15.84:0/3880406771 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f16580015d0 con 0x7f16601457f0 2024-03-29T10:36:22.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.442+0000 7f16567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:22.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.442+0000 7f16567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:22.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.442+0000 7f16567fc640 10 monclient: dump: 2024-03-29T10:36:22.440 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:22.440 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:22.441 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:22.441 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:22.441 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:22.441 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:22.441 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:22.441 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:22.441 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:22.441 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:22.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.442+0000 7f1657fff640 4 set_mon_vals no callback set 2024-03-29T10:36:22.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.443+0000 7f1667e88640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:22.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.443+0000 7f1667e88640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:22.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.443+0000 7f1667e88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:36:22.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.443+0000 7f1667e88640 1 -- 172.21.15.84:0/3880406771 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1648005080 con 0x7f16601457f0 2024-03-29T10:36:22.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.444+0000 7f16567fc640 1 -- 172.21.15.84:0/3880406771 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f165801ade0 con 0x7f16601457f0 2024-03-29T10:36:22.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.445+0000 7f16567fc640 1 -- 172.21.15.84:0/3880406771 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f1658084cf0 con 0x7f16601457f0 2024-03-29T10:36:22.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.445+0000 7f16567fc640 10 client.4282.objecter ms_dispatch 0x7f1660000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:36:22.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.445+0000 7f16567fc640 3 client.4282.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:36:22.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.445+0000 7f16567fc640 3 client.4282.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:36:22.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.445+0000 7f16567fc640 20 client.4282.objecter dump_active .. 0 homeless 2024-03-29T10:36:22.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.445+0000 7f1665c24640 10 monclient: get_auth_request con 0x7f164406de50 auth_method 0 2024-03-29T10:36:22.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.446+0000 7f16567fc640 10 client.4282.objecter ms_handle_connect 0x7f164406de50 2024-03-29T10:36:22.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.452+0000 7f16567fc640 1 -- 172.21.15.84:0/3880406771 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f16580505d0 con 0x7f16601457f0 2024-03-29T10:36:22.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.452+0000 7f16567fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:22.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.452+0000 7f16567fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:36:22.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.766+0000 7f1667e88640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:22.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.766+0000 7f1667e88640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:22.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.766+0000 7f1667e88640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:36:22.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.766+0000 7f1667e88640 1 -- 172.21.15.84:0/3880406771 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1648005330 con 0x7f16601457f0 2024-03-29T10:36:22.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.768+0000 7f16567fc640 1 -- 172.21.15.84:0/3880406771 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f1658052500 con 0x7f16601457f0 2024-03-29T10:36:22.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.768+0000 7f16567fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:22.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.768+0000 7f16567fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:36:22.755 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:36:22.755 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:36:22.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.771+0000 7f1667e88640 10 librados: watch_flush enter 2024-03-29T10:36:22.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.772+0000 7f1667e88640 10 librados: watch_flush exit 2024-03-29T10:36:22.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.772+0000 7f1667e88640 20 client.4282.objecter shutdown clearing up homeless session... 2024-03-29T10:36:22.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.772+0000 7f1667e88640 10 client.4282.objecter successfully canceled tick 2024-03-29T10:36:22.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.773+0000 7f1667e88640 1 -- 172.21.15.84:0/3880406771 >> v1:172.21.15.86:6816/3337225337 conn(0x7f164406de50 legacy=0x7f1644070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:22.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.773+0000 7f1667e88640 10 monclient: shutdown 2024-03-29T10:36:22.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.773+0000 7f1667e88640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:22.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.773+0000 7f1667e88640 1 -- 172.21.15.84:0/3880406771 >> v1:172.21.15.86:6789/0 conn(0x7f16601457f0 legacy=0x7f16601e2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:22.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.773+0000 7f1667e88640 1 -- 172.21.15.84:0/3880406771 shutdown_connections 2024-03-29T10:36:22.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.773+0000 7f1667e88640 1 -- 172.21.15.84:0/3880406771 >> 172.21.15.84:0/3880406771 conn(0x7f1660066a00 msgr2=0x7f16600b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:22.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.775+0000 7f1667e88640 1 -- 172.21.15.84:0/3880406771 shutdown_connections 2024-03-29T10:36:22.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.775+0000 7f1667e88640 1 -- 172.21.15.84:0/3880406771 wait complete. 2024-03-29T10:36:22.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:22.777+0000 7f1667e88640 1 librados: shutdown 2024-03-29T10:36:22.779 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:36:29.586 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:36:29.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.753+0000 7f0d1d053640 10 monclient: get_monmap_and_config 2024-03-29T10:36:29.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.753+0000 7f0d1d053640 10 monclient: build_initial_monmap 2024-03-29T10:36:29.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.753+0000 7f0d1d053640 10 monclient: monmap: 2024-03-29T10:36:29.738 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:36:29.738 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:29.738 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:36:29.754094+0000 2024-03-29T10:36:29.738 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:36:29.754094+0000 2024-03-29T10:36:29.738 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:29.738 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:29.738 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:36:29.739 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:36:29.739 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:36:29.739 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:29.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.754+0000 7f0d1d053640 1 Processor -- start 2024-03-29T10:36:29.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.755+0000 7f0d1d053640 1 -- start start 2024-03-29T10:36:29.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.755+0000 7f0d1d053640 10 monclient: init 2024-03-29T10:36:29.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.755+0000 7f0d1d053640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:29.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.755+0000 7f0d1d053640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:36:29.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.756+0000 7f0d1d053640 10 monclient(hunting): picked mon.noname-c con 0x7f0d100389e0 addr v1:172.21.15.86:6789/0 2024-03-29T10:36:29.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.756+0000 7f0d1d053640 10 monclient(hunting): picked mon.noname-b con 0x7f0d10038dc0 addr v1:172.21.15.84:6790/0 2024-03-29T10:36:29.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.756+0000 7f0d1d053640 10 monclient(hunting): picked mon.noname-a con 0x7f0d10030660 addr v1:172.21.15.84:6789/0 2024-03-29T10:36:29.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.756+0000 7f0d1d053640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d100c54c0 con 0x7f0d10030660 2024-03-29T10:36:29.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.756+0000 7f0d1d053640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d10036430 con 0x7f0d100389e0 2024-03-29T10:36:29.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.756+0000 7f0d1d053640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d10039840 con 0x7f0d10038dc0 2024-03-29T10:36:29.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.756+0000 7f0d1d053640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:29.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.756+0000 7f0d1d053640 10 monclient(hunting): authenticate will time out at 892.019201s 2024-03-29T10:36:29.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.759+0000 7f0d17fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f0d100389e0 0x7f0d10033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:44813/0 (socket says 172.21.15.84:44813) 2024-03-29T10:36:29.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.759+0000 7f0d17fff640 1 -- 172.21.15.84:0/1173090843 learned_addr learned my addr 172.21.15.84:0/1173090843 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:36:29.750 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 1 -- 172.21.15.84:0/1173090843 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2046392406 0 0) 0x7f0d10039840 con 0x7f0d10038dc0 2024-03-29T10:36:29.750 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:36:29.750 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:29.750 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:29.750 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 10 monclient(hunting): my global_id is 4298 2024-03-29T10:36:29.750 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 1 -- 172.21.15.84:0/1173090843 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d08003520 con 0x7f0d10038dc0 2024-03-29T10:36:29.750 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 1 -- 172.21.15.84:0/1173090843 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1189304455 0 0) 0x7f0d100c54c0 con 0x7f0d10030660 2024-03-29T10:36:29.750 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:36:29.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:29.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:29.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 10 monclient(hunting): my global_id is 4284 2024-03-29T10:36:29.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 1 -- 172.21.15.84:0/1173090843 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d10039840 con 0x7f0d10030660 2024-03-29T10:36:29.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 1 -- 172.21.15.84:0/1173090843 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3773782662 0 0) 0x7f0d08003520 con 0x7f0d10038dc0 2024-03-29T10:36:29.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 1 -- 172.21.15.84:0/1173090843 >> v1:172.21.15.86:6789/0 conn(0x7f0d100389e0 legacy=0x7f0d10033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:29.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 1 -- 172.21.15.84:0/1173090843 >> v1:172.21.15.84:6789/0 conn(0x7f0d10030660 legacy=0x7f0d100d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:29.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:36:29.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 1 monclient: found mon.noname-b 2024-03-29T10:36:29.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:29.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:36:29.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 1 -- 172.21.15.84:0/1173090843 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d10035690 con 0x7f0d10038dc0 2024-03-29T10:36:29.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 10 monclient: _finish_auth 0 2024-03-29T10:36:29.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:36:29.751 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:29.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 1 -- 172.21.15.84:0/1173090843 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0d04001690 con 0x7f0d10038dc0 2024-03-29T10:36:29.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:29.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:36:29.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 10 monclient: dump: 2024-03-29T10:36:29.752 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:29.752 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:29.752 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:29.752 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:29.752 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:29.752 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:29.752 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:29.752 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:29.752 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:29.752 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:29.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d1d053640 5 monclient: authenticate success, global_id 4298 2024-03-29T10:36:29.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d1d053640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:29.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 1 -- 172.21.15.84:0/1173090843 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d04001860 con 0x7f0d10038dc0 2024-03-29T10:36:29.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:29.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 1 -- 172.21.15.84:0/1173090843 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0d04001cb0 con 0x7f0d10038dc0 2024-03-29T10:36:29.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:29.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:29.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d16ffd640 10 monclient: dump: 2024-03-29T10:36:29.753 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:29.753 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:29.753 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:29.753 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:29.753 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:29.753 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:29.753 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:29.753 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:29.754 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:29.754 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:29.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d1d053640 10 monclient: get_monmap_and_config success 2024-03-29T10:36:29.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.760+0000 7f0d1d053640 4 set_mon_vals no callback set 2024-03-29T10:36:29.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.761+0000 7f0d1d053640 10 monclient: shutdown 2024-03-29T10:36:29.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.761+0000 7f0d1d053640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:29.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.761+0000 7f0d1d053640 1 -- 172.21.15.84:0/1173090843 >> v1:172.21.15.84:6790/0 conn(0x7f0d10038dc0 legacy=0x7f0d100cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:29.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.762+0000 7f0d1d053640 1 -- 172.21.15.84:0/1173090843 shutdown_connections 2024-03-29T10:36:29.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.762+0000 7f0d1d053640 1 -- 172.21.15.84:0/1173090843 >> 172.21.15.84:0/1173090843 conn(0x7f0d10005c00 msgr2=0x7f0d10006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:29.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.764+0000 7f0d1d053640 1 -- 172.21.15.84:0/1173090843 shutdown_connections 2024-03-29T10:36:29.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.764+0000 7f0d1d053640 1 -- 172.21.15.84:0/1173090843 wait complete. 2024-03-29T10:36:29.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.765+0000 7f0d1d053640 10 monclient: build_initial_monmap 2024-03-29T10:36:29.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.765+0000 7f0d1d053640 10 monclient: monmap: 2024-03-29T10:36:29.754 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:36:29.755 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:36:29.755 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:36:29.755 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:36:29.755 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:29.755 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:29.755 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:36:29.755 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:36:29.755 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:36:29.755 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:29.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.765+0000 7f0d1d053640 1 librados: starting msgr at 2024-03-29T10:36:29.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.765+0000 7f0d1d053640 1 librados: starting objecter 2024-03-29T10:36:29.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.765+0000 7f0d1d053640 1 Processor -- start 2024-03-29T10:36:29.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.766+0000 7f0d1d053640 1 -- start start 2024-03-29T10:36:29.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.766+0000 7f0d1d053640 1 librados: setting wanted keys 2024-03-29T10:36:29.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.766+0000 7f0d1d053640 1 librados: calling monclient init 2024-03-29T10:36:29.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.766+0000 7f0d1d053640 10 monclient: init 2024-03-29T10:36:29.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.767+0000 7f0d1d053640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:29.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.767+0000 7f0d1d053640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:36:29.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.767+0000 7f0d1d053640 10 monclient(hunting): picked mon.noname-a con 0x7f0d10030660 addr v1:172.21.15.84:6789/0 2024-03-29T10:36:29.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.767+0000 7f0d1d053640 10 monclient(hunting): picked mon.noname-b con 0x7f0d100389e0 addr v1:172.21.15.86:6789/0 2024-03-29T10:36:29.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.767+0000 7f0d1d053640 10 monclient(hunting): picked mon.noname-c con 0x7f0d1002fd20 addr v1:172.21.15.84:6790/0 2024-03-29T10:36:29.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.767+0000 7f0d1d053640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d1002f5b0 con 0x7f0d10030660 2024-03-29T10:36:29.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.767+0000 7f0d1d053640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d1000a220 con 0x7f0d100389e0 2024-03-29T10:36:29.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.767+0000 7f0d1d053640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d101610c0 con 0x7f0d1002fd20 2024-03-29T10:36:29.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.767+0000 7f0d1d053640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:29.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.767+0000 7f0d1d053640 10 monclient(hunting): authenticate will time out at 892.030351s 2024-03-29T10:36:29.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.772+0000 7f0d177fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f0d100389e0 0x7f0d1000cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:44889/0 (socket says 172.21.15.84:44889) 2024-03-29T10:36:29.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.772+0000 7f0d177fe640 1 -- 172.21.15.84:0/3888198748 learned_addr learned my addr 172.21.15.84:0/3888198748 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:36:29.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.773+0000 7f0d14ff9640 10 client.?.objecter ms_handle_connect 0x7f0d100389e0 2024-03-29T10:36:29.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.773+0000 7f0d14ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:29.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.774+0000 7f0d14ff9640 1 -- 172.21.15.84:0/3888198748 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561182090 0 0) 0x7f0d1000a220 con 0x7f0d100389e0 2024-03-29T10:36:29.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.774+0000 7f0d14ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:36:29.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.774+0000 7f0d14ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:29.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.774+0000 7f0d14ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:29.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.774+0000 7f0d14ff9640 10 monclient(hunting): my global_id is 4288 2024-03-29T10:36:29.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.774+0000 7f0d14ff9640 1 -- 172.21.15.84:0/3888198748 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cfc004eb0 con 0x7f0d100389e0 2024-03-29T10:36:29.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.774+0000 7f0d14ff9640 10 client.?.objecter ms_handle_connect 0x7f0d1002fd20 2024-03-29T10:36:29.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.774+0000 7f0d14ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:29.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.775+0000 7f0d14ff9640 10 client.?.objecter ms_handle_connect 0x7f0d10030660 2024-03-29T10:36:29.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.775+0000 7f0d14ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:29.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.775+0000 7f0d14ff9640 1 -- 172.21.15.84:0/3888198748 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 632083930 0 0) 0x7f0d101610c0 con 0x7f0d1002fd20 2024-03-29T10:36:29.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.775+0000 7f0d14ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:36:29.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.775+0000 7f0d14ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:29.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.775+0000 7f0d14ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:29.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.775+0000 7f0d14ff9640 10 monclient(hunting): my global_id is 4301 2024-03-29T10:36:29.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.775+0000 7f0d14ff9640 1 -- 172.21.15.84:0/3888198748 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d1000a220 con 0x7f0d1002fd20 2024-03-29T10:36:29.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.775+0000 7f0d14ff9640 1 -- 172.21.15.84:0/3888198748 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1049557571 0 0) 0x7f0cfc004eb0 con 0x7f0d100389e0 2024-03-29T10:36:29.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.775+0000 7f0d14ff9640 1 -- 172.21.15.84:0/3888198748 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d101610c0 con 0x7f0d100389e0 2024-03-29T10:36:29.764 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.775+0000 7f0d14ff9640 1 -- 172.21.15.84:0/3888198748 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0d04001c30 con 0x7f0d100389e0 2024-03-29T10:36:29.764 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.776+0000 7f0d14ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:29.764 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.776+0000 7f0d14ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:36:29.764 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.776+0000 7f0d14ff9640 10 monclient(hunting): dump: 2024-03-29T10:36:29.764 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:29.764 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:29.764 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:29.764 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:29.764 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:29.764 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:29.764 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:29.764 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:29.764 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:29.764 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:29.764 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.776+0000 7f0d14ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:36:29.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.776+0000 7f0d14ff9640 1 -- 172.21.15.84:0/3888198748 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3552399720 0 0) 0x7f0d1000a220 con 0x7f0d1002fd20 2024-03-29T10:36:29.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.776+0000 7f0d14ff9640 1 -- 172.21.15.84:0/3888198748 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cfc004eb0 con 0x7f0d1002fd20 2024-03-29T10:36:29.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.776+0000 7f0d14ff9640 1 -- 172.21.15.84:0/3888198748 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212007572 0 0) 0x7f0d1002f5b0 con 0x7f0d10030660 2024-03-29T10:36:29.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.776+0000 7f0d14ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:36:29.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.776+0000 7f0d14ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:29.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.776+0000 7f0d14ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:29.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.776+0000 7f0d14ff9640 10 monclient(hunting): my global_id is 4287 2024-03-29T10:36:29.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.776+0000 7f0d14ff9640 1 -- 172.21.15.84:0/3888198748 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d1000a220 con 0x7f0d10030660 2024-03-29T10:36:29.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.776+0000 7f0d14ff9640 1 -- 172.21.15.84:0/3888198748 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0d1804fdd0 con 0x7f0d1002fd20 2024-03-29T10:36:29.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.776+0000 7f0d14ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:29.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.776+0000 7f0d14ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:29.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.776+0000 7f0d14ff9640 10 monclient(hunting): dump: 2024-03-29T10:36:29.765 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:29.765 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:29.765 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:29.766 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:29.766 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:29.766 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:29.766 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:29.766 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:29.766 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:29.766 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:29.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.777+0000 7f0d14ff9640 1 -- 172.21.15.84:0/3888198748 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 298018493 0 0) 0x7f0d101610c0 con 0x7f0d100389e0 2024-03-29T10:36:29.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.777+0000 7f0d14ff9640 1 -- 172.21.15.84:0/3888198748 >> v1:172.21.15.84:6790/0 conn(0x7f0d1002fd20 legacy=0x7f0d1000d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:29.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.777+0000 7f0d14ff9640 1 -- 172.21.15.84:0/3888198748 >> v1:172.21.15.84:6789/0 conn(0x7f0d10030660 legacy=0x7f0d101609b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:29.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.778+0000 7f0d14ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:36:29.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.778+0000 7f0d14ff9640 1 monclient: found mon.b 2024-03-29T10:36:29.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.778+0000 7f0d14ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:29.766 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.778+0000 7f0d14ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:36:29.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.778+0000 7f0d14ff9640 1 -- 172.21.15.84:0/3888198748 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d1002f800 con 0x7f0d100389e0 2024-03-29T10:36:29.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.778+0000 7f0d14ff9640 10 monclient: _finish_auth 0 2024-03-29T10:36:29.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.778+0000 7f0d14ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:36:29.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.778+0000 7f0d14ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:29.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.779+0000 7f0d1d053640 5 monclient: authenticate success, global_id 4288 2024-03-29T10:36:29.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.779+0000 7f0d1d053640 10 monclient: _renew_subs 2024-03-29T10:36:29.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.779+0000 7f0d1d053640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:36:29.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.779+0000 7f0d1d053640 1 -- 172.21.15.84:0/3888198748 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d100c7bf0 con 0x7f0d100389e0 2024-03-29T10:36:29.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.779+0000 7f0d1d053640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:36:29.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.779+0000 7f0d1d053640 10 monclient: _renew_subs 2024-03-29T10:36:29.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.779+0000 7f0d1d053640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:36:29.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.779+0000 7f0d1d053640 1 -- 172.21.15.84:0/3888198748 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d1002eff0 con 0x7f0d100389e0 2024-03-29T10:36:29.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.779+0000 7f0d1d053640 1 librados: init done 2024-03-29T10:36:29.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.779+0000 7f0d14ff9640 1 -- 172.21.15.84:0/3888198748 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d040038a0 con 0x7f0d100389e0 2024-03-29T10:36:29.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.779+0000 7f0d14ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:29.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.779+0000 7f0d14ff9640 1 -- 172.21.15.84:0/3888198748 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0d04003cf0 con 0x7f0d100389e0 2024-03-29T10:36:29.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.779+0000 7f0d14ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:29.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.779+0000 7f0d14ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:29.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.779+0000 7f0d14ff9640 10 monclient: dump: 2024-03-29T10:36:29.770 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:29.770 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:29.770 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:29.770 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:29.770 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:29.770 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:29.770 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:29.770 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:29.771 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:29.771 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:29.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.779+0000 7f0d15ffb640 4 set_mon_vals no callback set 2024-03-29T10:36:29.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.781+0000 7f0d1d053640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:29.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.781+0000 7f0d1d053640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:29.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.781+0000 7f0d1d053640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:36:29.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.781+0000 7f0d1d053640 1 -- 172.21.15.84:0/3888198748 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf8005080 con 0x7f0d100389e0 2024-03-29T10:36:29.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.781+0000 7f0d14ff9640 1 -- 172.21.15.84:0/3888198748 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f0d04003e50 con 0x7f0d100389e0 2024-03-29T10:36:29.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.782+0000 7f0d14ff9640 1 -- 172.21.15.84:0/3888198748 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f0d040548c0 con 0x7f0d100389e0 2024-03-29T10:36:29.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.782+0000 7f0d14ff9640 10 client.4288.objecter ms_dispatch 0x7f0d18000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:36:29.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.782+0000 7f0d14ff9640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:36:29.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.783+0000 7f0d14ff9640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:36:29.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.783+0000 7f0d14ff9640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-29T10:36:29.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.783+0000 7f0d17fff640 10 monclient: get_auth_request con 0x7f0cfc06dbb0 auth_method 0 2024-03-29T10:36:29.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.784+0000 7f0d14ff9640 10 client.4288.objecter ms_handle_connect 0x7f0cfc06dbb0 2024-03-29T10:36:29.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.790+0000 7f0d14ff9640 1 -- 172.21.15.84:0/3888198748 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0d04050b60 con 0x7f0d100389e0 2024-03-29T10:36:29.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.790+0000 7f0d14ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:29.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:29.790+0000 7f0d14ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:36:30.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:30.118+0000 7f0d1d053640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:30.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:30.118+0000 7f0d1d053640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:30.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:30.118+0000 7f0d1d053640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:36:30.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:30.118+0000 7f0d1d053640 1 -- 172.21.15.84:0/3888198748 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0cf8005330 con 0x7f0d100389e0 2024-03-29T10:36:30.108 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:36:30.108 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:36:30.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:30.122+0000 7f0d14ff9640 1 -- 172.21.15.84:0/3888198748 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f0d04050b60 con 0x7f0d100389e0 2024-03-29T10:36:30.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:30.122+0000 7f0d14ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:30.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:30.122+0000 7f0d14ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:36:30.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:30.126+0000 7f0d1d053640 10 librados: watch_flush enter 2024-03-29T10:36:30.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:30.127+0000 7f0d1d053640 10 librados: watch_flush exit 2024-03-29T10:36:30.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:30.127+0000 7f0d1d053640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-29T10:36:30.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:30.127+0000 7f0d1d053640 10 client.4288.objecter successfully canceled tick 2024-03-29T10:36:30.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:30.127+0000 7f0d1d053640 1 -- 172.21.15.84:0/3888198748 >> v1:172.21.15.86:6816/3337225337 conn(0x7f0cfc06dbb0 legacy=0x7f0cfc070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:30.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:30.127+0000 7f0d1d053640 10 monclient: shutdown 2024-03-29T10:36:30.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:30.127+0000 7f0d1d053640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:30.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:30.127+0000 7f0d1d053640 1 -- 172.21.15.84:0/3888198748 >> v1:172.21.15.86:6789/0 conn(0x7f0d100389e0 legacy=0x7f0d1000cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:30.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:30.127+0000 7f0d1d053640 1 -- 172.21.15.84:0/3888198748 shutdown_connections 2024-03-29T10:36:30.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:30.127+0000 7f0d1d053640 1 -- 172.21.15.84:0/3888198748 >> 172.21.15.84:0/3888198748 conn(0x7f0d10005c00 msgr2=0x7f0d10034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:30.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:30.128+0000 7f0d1d053640 1 -- 172.21.15.84:0/3888198748 shutdown_connections 2024-03-29T10:36:30.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:30.128+0000 7f0d1d053640 1 -- 172.21.15.84:0/3888198748 wait complete. 2024-03-29T10:36:30.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:30.129+0000 7f0d1d053640 1 librados: shutdown 2024-03-29T10:36:30.133 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:36:36.940 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:36:37.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.103+0000 7f849a966640 10 monclient: get_monmap_and_config 2024-03-29T10:36:37.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.103+0000 7f849a966640 10 monclient: build_initial_monmap 2024-03-29T10:36:37.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.103+0000 7f849a966640 10 monclient: monmap: 2024-03-29T10:36:37.091 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:36:37.092 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:37.092 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:36:37.104824+0000 2024-03-29T10:36:37.092 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:36:37.104824+0000 2024-03-29T10:36:37.092 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:37.092 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:37.092 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:36:37.092 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:36:37.092 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:36:37.092 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:37.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.106+0000 7f849a966640 1 Processor -- start 2024-03-29T10:36:37.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.106+0000 7f849a966640 1 -- start start 2024-03-29T10:36:37.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.106+0000 7f849a966640 10 monclient: init 2024-03-29T10:36:37.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.107+0000 7f849a966640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:37.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.107+0000 7f849a966640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:36:37.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.107+0000 7f849a966640 10 monclient(hunting): picked mon.noname-b con 0x7f848c034850 addr v1:172.21.15.84:6790/0 2024-03-29T10:36:37.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.111+0000 7f849a966640 10 monclient(hunting): picked mon.noname-a con 0x7f848c0c51c0 addr v1:172.21.15.84:6789/0 2024-03-29T10:36:37.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.111+0000 7f849a966640 10 monclient(hunting): picked mon.noname-c con 0x7f848c0c7cd0 addr v1:172.21.15.86:6789/0 2024-03-29T10:36:37.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.111+0000 7f8499163640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f848c0c51c0 0x7f848c029800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:60687/0 (socket says 172.21.15.84:60687) 2024-03-29T10:36:37.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.111+0000 7f8499163640 1 -- 172.21.15.84:0/279912013 learned_addr learned my addr 172.21.15.84:0/279912013 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:36:37.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.111+0000 7f849a966640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f848c029190 con 0x7f848c0c51c0 2024-03-29T10:36:37.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.111+0000 7f849a966640 1 -- 172.21.15.84:0/279912013 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f848c0293e0 con 0x7f848c0c7cd0 2024-03-29T10:36:37.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.111+0000 7f849a966640 1 -- 172.21.15.84:0/279912013 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f848c0c8340 con 0x7f848c034850 2024-03-29T10:36:37.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.111+0000 7f849a966640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:37.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.111+0000 7f849a966640 10 monclient(hunting): authenticate will time out at 899.374873s 2024-03-29T10:36:37.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.112+0000 7f8498962640 1 -- 172.21.15.84:0/279912013 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506932721 0 0) 0x7f848c029190 con 0x7f848c0c51c0 2024-03-29T10:36:37.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.112+0000 7f8498962640 10 monclient(hunting): _negotiate 2024-03-29T10:36:37.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.112+0000 7f8498962640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:37.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.112+0000 7f8498962640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:37.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.112+0000 7f8498962640 10 monclient(hunting): my global_id is 4290 2024-03-29T10:36:37.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.112+0000 7f8498962640 1 -- 172.21.15.84:0/279912013 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8480003520 con 0x7f848c0c51c0 2024-03-29T10:36:37.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.115+0000 7f8498962640 1 -- 172.21.15.84:0/279912013 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3211371851 0 0) 0x7f8480003520 con 0x7f848c0c51c0 2024-03-29T10:36:37.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.115+0000 7f8498962640 1 -- 172.21.15.84:0/279912013 >> v1:172.21.15.84:6790/0 conn(0x7f848c034850 legacy=0x7f848c0c4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:36:37.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.115+0000 7f8498962640 1 -- 172.21.15.84:0/279912013 >> v1:172.21.15.86:6789/0 conn(0x7f848c0c7cd0 legacy=0x7f848c029f10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:36:37.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.116+0000 7f8498962640 10 monclient: _finish_hunting 0 2024-03-29T10:36:37.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.116+0000 7f8498962640 1 monclient: found mon.noname-a 2024-03-29T10:36:37.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.116+0000 7f8498962640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:37.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.116+0000 7f8498962640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T10:36:37.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.116+0000 7f8498962640 1 -- 172.21.15.84:0/279912013 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f848c032980 con 0x7f848c0c51c0 2024-03-29T10:36:37.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.116+0000 7f8498962640 10 monclient: _finish_auth 0 2024-03-29T10:36:37.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.116+0000 7f8498962640 10 monclient: _check_auth_tickets 2024-03-29T10:36:37.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.116+0000 7f8498962640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:37.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.116+0000 7f8498962640 1 -- 172.21.15.84:0/279912013 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f8490001690 con 0x7f848c0c51c0 2024-03-29T10:36:37.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.116+0000 7f8498962640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:37.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.116+0000 7f8498962640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:36:37.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.116+0000 7f8498962640 10 monclient: dump: 2024-03-29T10:36:37.102 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:37.102 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:37.102 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:37.102 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:37.102 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:37.102 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:37.102 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:37.102 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:37.102 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:37.102 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:37.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.116+0000 7f849a966640 5 monclient: authenticate success, global_id 4290 2024-03-29T10:36:37.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.116+0000 7f849a966640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:37.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.120+0000 7f8498962640 1 -- 172.21.15.84:0/279912013 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8490001860 con 0x7f848c0c51c0 2024-03-29T10:36:37.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.120+0000 7f8498962640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:37.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.120+0000 7f8498962640 1 -- 172.21.15.84:0/279912013 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f8490001cb0 con 0x7f848c0c51c0 2024-03-29T10:36:37.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.120+0000 7f8498962640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:37.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.120+0000 7f8498962640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:36:37.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.120+0000 7f8498962640 10 monclient: dump: 2024-03-29T10:36:37.105 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:37.105 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:37.105 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:37.106 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:37.106 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:37.106 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:37.106 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:37.106 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:37.106 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:37.106 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:37.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.120+0000 7f849a966640 10 monclient: get_monmap_and_config success 2024-03-29T10:36:37.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.120+0000 7f849a966640 4 set_mon_vals no callback set 2024-03-29T10:36:37.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.121+0000 7f849a966640 10 monclient: shutdown 2024-03-29T10:36:37.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.121+0000 7f849a966640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:37.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.121+0000 7f849a966640 1 -- 172.21.15.84:0/279912013 >> v1:172.21.15.84:6789/0 conn(0x7f848c0c51c0 legacy=0x7f848c029800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:37.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.124+0000 7f849a966640 1 -- 172.21.15.84:0/279912013 shutdown_connections 2024-03-29T10:36:37.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.124+0000 7f849a966640 1 -- 172.21.15.84:0/279912013 >> 172.21.15.84:0/279912013 conn(0x7f848c0058f0 msgr2=0x7f848c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:37.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.125+0000 7f849a966640 1 -- 172.21.15.84:0/279912013 shutdown_connections 2024-03-29T10:36:37.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.125+0000 7f849a966640 1 -- 172.21.15.84:0/279912013 wait complete. 2024-03-29T10:36:37.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.126+0000 7f849a966640 10 monclient: build_initial_monmap 2024-03-29T10:36:37.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.126+0000 7f849a966640 10 monclient: monmap: 2024-03-29T10:36:37.112 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:36:37.112 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:36:37.112 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:36:37.112 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:36:37.112 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:37.112 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:37.112 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:36:37.113 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:36:37.113 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:36:37.113 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:37.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.127+0000 7f849a966640 1 librados: starting msgr at 2024-03-29T10:36:37.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.127+0000 7f849a966640 1 librados: starting objecter 2024-03-29T10:36:37.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.127+0000 7f849a966640 1 Processor -- start 2024-03-29T10:36:37.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.127+0000 7f849a966640 1 -- start start 2024-03-29T10:36:37.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.127+0000 7f849a966640 1 librados: setting wanted keys 2024-03-29T10:36:37.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.127+0000 7f849a966640 1 librados: calling monclient init 2024-03-29T10:36:37.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.127+0000 7f849a966640 10 monclient: init 2024-03-29T10:36:37.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.128+0000 7f849a966640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:37.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.128+0000 7f849a966640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:36:37.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.129+0000 7f849a966640 10 monclient(hunting): picked mon.noname-a con 0x7f848c034850 addr v1:172.21.15.84:6789/0 2024-03-29T10:36:37.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.129+0000 7f849a966640 10 monclient(hunting): picked mon.noname-c con 0x7f848c0c51c0 addr v1:172.21.15.84:6790/0 2024-03-29T10:36:37.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.129+0000 7f849a966640 10 monclient(hunting): picked mon.noname-b con 0x7f848c0c7cd0 addr v1:172.21.15.86:6789/0 2024-03-29T10:36:37.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.129+0000 7f849a966640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f848c0cb1d0 con 0x7f848c034850 2024-03-29T10:36:37.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.129+0000 7f849a966640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f848c0cb4b0 con 0x7f848c0c7cd0 2024-03-29T10:36:37.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.129+0000 7f849a966640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f848c0cb790 con 0x7f848c0c51c0 2024-03-29T10:36:37.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.129+0000 7f849a966640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:37.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.129+0000 7f849a966640 10 monclient(hunting): authenticate will time out at 899.392793s 2024-03-29T10:36:37.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.133+0000 7f8499964640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f848c034850 0x7f848c0b6d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:32841/0 (socket says 172.21.15.84:32841) 2024-03-29T10:36:37.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.133+0000 7f8499964640 1 -- 172.21.15.84:0/505718218 learned_addr learned my addr 172.21.15.84:0/505718218 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:36:37.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.133+0000 7f848a7fc640 10 client.?.objecter ms_handle_connect 0x7f848c034850 2024-03-29T10:36:37.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.133+0000 7f848a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:37.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.133+0000 7f848a7fc640 10 client.?.objecter ms_handle_connect 0x7f848c0c51c0 2024-03-29T10:36:37.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.133+0000 7f848a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:37.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.134+0000 7f848a7fc640 1 -- 172.21.15.84:0/505718218 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2349348670 0 0) 0x7f848c0cb1d0 con 0x7f848c034850 2024-03-29T10:36:37.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.134+0000 7f848a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:36:37.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.134+0000 7f848a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:37.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.134+0000 7f848a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:37.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.134+0000 7f848a7fc640 10 monclient(hunting): my global_id is 4293 2024-03-29T10:36:37.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.134+0000 7f848a7fc640 1 -- 172.21.15.84:0/505718218 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8478004eb0 con 0x7f848c034850 2024-03-29T10:36:37.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.134+0000 7f848a7fc640 1 -- 172.21.15.84:0/505718218 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484393211 0 0) 0x7f848c0cb790 con 0x7f848c0c51c0 2024-03-29T10:36:37.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.134+0000 7f848a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:36:37.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.134+0000 7f848a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:37.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.134+0000 7f848a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:37.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.134+0000 7f848a7fc640 10 monclient(hunting): my global_id is 4304 2024-03-29T10:36:37.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.134+0000 7f848a7fc640 1 -- 172.21.15.84:0/505718218 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f848c0cb1d0 con 0x7f848c0c51c0 2024-03-29T10:36:37.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.134+0000 7f848a7fc640 1 -- 172.21.15.84:0/505718218 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 232972158 0 0) 0x7f8478004eb0 con 0x7f848c034850 2024-03-29T10:36:37.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.134+0000 7f848a7fc640 1 -- 172.21.15.84:0/505718218 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f848c0cb790 con 0x7f848c034850 2024-03-29T10:36:37.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.134+0000 7f848a7fc640 1 -- 172.21.15.84:0/505718218 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f8484004080 con 0x7f848c034850 2024-03-29T10:36:37.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.134+0000 7f848a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:37.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.134+0000 7f848a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:36:37.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.135+0000 7f848a7fc640 10 monclient(hunting): dump: 2024-03-29T10:36:37.124 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:37.124 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:37.124 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:37.124 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:37.125 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:37.125 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:37.125 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:37.125 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:37.125 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:37.125 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:37.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.135+0000 7f848a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:36:37.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.135+0000 7f848a7fc640 1 -- 172.21.15.84:0/505718218 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2657250922 0 0) 0x7f848c0cb1d0 con 0x7f848c0c51c0 2024-03-29T10:36:37.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.135+0000 7f848a7fc640 1 -- 172.21.15.84:0/505718218 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8478004eb0 con 0x7f848c0c51c0 2024-03-29T10:36:37.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.135+0000 7f848a7fc640 1 -- 172.21.15.84:0/505718218 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f8490001b70 con 0x7f848c0c51c0 2024-03-29T10:36:37.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.135+0000 7f848a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:37.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.135+0000 7f848a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:37.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.135+0000 7f848a7fc640 10 monclient(hunting): dump: 2024-03-29T10:36:37.125 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:37.126 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:37.126 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:37.126 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:37.126 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:37.126 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:37.126 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:37.126 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:37.126 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:37.126 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:37.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.135+0000 7f848a7fc640 1 -- 172.21.15.84:0/505718218 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 537299267 0 0) 0x7f8478004eb0 con 0x7f848c0c51c0 2024-03-29T10:36:37.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.135+0000 7f848a7fc640 1 -- 172.21.15.84:0/505718218 >> v1:172.21.15.86:6789/0 conn(0x7f848c0c7cd0 legacy=0x7f848c167900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:36:37.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.136+0000 7f848a7fc640 1 -- 172.21.15.84:0/505718218 >> v1:172.21.15.84:6789/0 conn(0x7f848c034850 legacy=0x7f848c0b6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:37.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.136+0000 7f848a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:36:37.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.136+0000 7f848a7fc640 1 monclient: found mon.c 2024-03-29T10:36:37.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.136+0000 7f848a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:37.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.136+0000 7f848a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:36:37.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.136+0000 7f848a7fc640 1 -- 172.21.15.84:0/505718218 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f848c0ca3a0 con 0x7f848c0c51c0 2024-03-29T10:36:37.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.136+0000 7f848a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:36:37.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.136+0000 7f848a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:36:37.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.136+0000 7f848a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:37.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.137+0000 7f848a7fc640 1 -- 172.21.15.84:0/505718218 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8490004bf0 con 0x7f848c0c51c0 2024-03-29T10:36:37.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.137+0000 7f848a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:37.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.137+0000 7f848a7fc640 1 -- 172.21.15.84:0/505718218 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f8490004f00 con 0x7f848c0c51c0 2024-03-29T10:36:37.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.137+0000 7f848a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:37.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.137+0000 7f848a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:37.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.137+0000 7f848a7fc640 10 monclient: dump: 2024-03-29T10:36:37.127 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:37.127 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:37.127 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:37.128 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:37.128 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:37.128 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:37.128 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:37.128 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:37.128 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:37.128 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:37.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.137+0000 7f849a966640 5 monclient: authenticate success, global_id 4304 2024-03-29T10:36:37.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.137+0000 7f849a966640 10 monclient: _renew_subs 2024-03-29T10:36:37.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.137+0000 7f849a966640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:36:37.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.137+0000 7f849a966640 1 -- 172.21.15.84:0/505718218 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f848c0c9f50 con 0x7f848c0c51c0 2024-03-29T10:36:37.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.137+0000 7f849a966640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:36:37.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.138+0000 7f848bfff640 4 set_mon_vals no callback set 2024-03-29T10:36:37.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.139+0000 7f848a7fc640 1 -- 172.21.15.84:0/505718218 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f849001aed0 con 0x7f848c0c51c0 2024-03-29T10:36:37.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.140+0000 7f849a966640 10 monclient: _renew_subs 2024-03-29T10:36:37.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.140+0000 7f849a966640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:36:37.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.140+0000 7f849a966640 1 -- 172.21.15.84:0/505718218 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f848c0c9580 con 0x7f848c0c51c0 2024-03-29T10:36:37.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.140+0000 7f849a966640 1 librados: init done 2024-03-29T10:36:37.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.142+0000 7f8477fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:37.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.142+0000 7f8477fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:37.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.142+0000 7f8477fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:36:37.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.142+0000 7f8477fff640 1 -- 172.21.15.84:0/505718218 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8470005080 con 0x7f848c0c51c0 2024-03-29T10:36:37.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.142+0000 7f848a7fc640 1 -- 172.21.15.84:0/505718218 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f8490054b40 con 0x7f848c0c51c0 2024-03-29T10:36:37.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.142+0000 7f848a7fc640 10 client.4304.objecter ms_dispatch 0x7f8494000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:36:37.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.142+0000 7f848a7fc640 3 client.4304.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:36:37.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.142+0000 7f848a7fc640 3 client.4304.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:36:37.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.142+0000 7f848a7fc640 20 client.4304.objecter dump_active .. 0 homeless 2024-03-29T10:36:37.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.143+0000 7f8499964640 10 monclient: get_auth_request con 0x7f847806ddf0 auth_method 0 2024-03-29T10:36:37.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.145+0000 7f848a7fc640 10 client.4304.objecter ms_handle_connect 0x7f847806ddf0 2024-03-29T10:36:37.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.153+0000 7f848a7fc640 1 -- 172.21.15.84:0/505718218 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8490050650 con 0x7f848c0c51c0 2024-03-29T10:36:37.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.153+0000 7f848a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:37.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.154+0000 7f848a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:36:37.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.478+0000 7f8477fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:37.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.478+0000 7f8477fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:37.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.478+0000 7f8477fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:36:37.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.478+0000 7f8477fff640 1 -- 172.21.15.84:0/505718218 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8470005330 con 0x7f848c0c51c0 2024-03-29T10:36:37.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.479+0000 7f848a7fc640 1 -- 172.21.15.84:0/505718218 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f8490052580 con 0x7f848c0c51c0 2024-03-29T10:36:37.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.480+0000 7f848a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:37.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.480+0000 7f848a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:36:37.466 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:36:37.466 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:36:37.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.483+0000 7f8477fff640 10 librados: watch_flush enter 2024-03-29T10:36:37.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.483+0000 7f8477fff640 10 librados: watch_flush exit 2024-03-29T10:36:37.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.483+0000 7f8477fff640 20 client.4304.objecter shutdown clearing up homeless session... 2024-03-29T10:36:37.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.483+0000 7f8477fff640 10 client.4304.objecter successfully canceled tick 2024-03-29T10:36:37.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.484+0000 7f8477fff640 1 -- 172.21.15.84:0/505718218 >> v1:172.21.15.86:6816/3337225337 conn(0x7f847806ddf0 legacy=0x7f84780702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:37.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.484+0000 7f8477fff640 10 monclient: shutdown 2024-03-29T10:36:37.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.484+0000 7f8477fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:37.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.484+0000 7f8477fff640 1 -- 172.21.15.84:0/505718218 >> v1:172.21.15.84:6790/0 conn(0x7f848c0c51c0 legacy=0x7f848c0b7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:37.471 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.486+0000 7f8477fff640 1 -- 172.21.15.84:0/505718218 shutdown_connections 2024-03-29T10:36:37.471 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.486+0000 7f8477fff640 1 -- 172.21.15.84:0/505718218 >> 172.21.15.84:0/505718218 conn(0x7f848c0058f0 msgr2=0x7f848c0c4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:37.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.488+0000 7f8477fff640 1 -- 172.21.15.84:0/505718218 shutdown_connections 2024-03-29T10:36:37.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.488+0000 7f8477fff640 1 -- 172.21.15.84:0/505718218 wait complete. 2024-03-29T10:36:37.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:37.488+0000 7f8477fff640 1 librados: shutdown 2024-03-29T10:36:37.489 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:36:44.296 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:36:44.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.467+0000 7f6d1192d640 10 monclient: get_monmap_and_config 2024-03-29T10:36:44.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.467+0000 7f6d1192d640 10 monclient: build_initial_monmap 2024-03-29T10:36:44.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.467+0000 7f6d1192d640 10 monclient: monmap: 2024-03-29T10:36:44.453 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:36:44.453 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:44.453 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:36:44.469076+0000 2024-03-29T10:36:44.453 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:36:44.469076+0000 2024-03-29T10:36:44.453 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:44.454 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:44.454 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:36:44.454 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:36:44.454 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:36:44.454 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:44.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.468+0000 7f6d1192d640 1 Processor -- start 2024-03-29T10:36:44.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.471+0000 7f6d1192d640 1 -- start start 2024-03-29T10:36:44.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.471+0000 7f6d1192d640 10 monclient: init 2024-03-29T10:36:44.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.472+0000 7f6d1192d640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:44.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.472+0000 7f6d1192d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:36:44.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.472+0000 7f6d1192d640 10 monclient(hunting): picked mon.noname-b con 0x7f6d040389e0 addr v1:172.21.15.84:6790/0 2024-03-29T10:36:44.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.472+0000 7f6d1192d640 10 monclient(hunting): picked mon.noname-c con 0x7f6d04038dc0 addr v1:172.21.15.86:6789/0 2024-03-29T10:36:44.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.472+0000 7f6d1192d640 10 monclient(hunting): picked mon.noname-a con 0x7f6d04030660 addr v1:172.21.15.84:6789/0 2024-03-29T10:36:44.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.472+0000 7f6d1192d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d040c54c0 con 0x7f6d04030660 2024-03-29T10:36:44.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.472+0000 7f6d1192d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d04036430 con 0x7f6d04038dc0 2024-03-29T10:36:44.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.472+0000 7f6d1192d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d04039840 con 0x7f6d040389e0 2024-03-29T10:36:44.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.472+0000 7f6d1192d640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:44.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.472+0000 7f6d1192d640 10 monclient(hunting): authenticate will time out at 906.736717s 2024-03-29T10:36:44.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.476+0000 7f6d1092b640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f6d040389e0 0x7f6d04033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:32827/0 (socket says 172.21.15.84:32827) 2024-03-29T10:36:44.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.476+0000 7f6d1092b640 1 -- 172.21.15.84:0/36547569 learned_addr learned my addr 172.21.15.84:0/36547569 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:36:44.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.477+0000 7f6d0b7fe640 1 -- 172.21.15.84:0/36547569 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4057593266 0 0) 0x7f6d04039840 con 0x7f6d040389e0 2024-03-29T10:36:44.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.477+0000 7f6d0b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:44.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.477+0000 7f6d0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:44.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.477+0000 7f6d0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:44.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.478+0000 7f6d0b7fe640 10 monclient(hunting): my global_id is 4307 2024-03-29T10:36:44.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.478+0000 7f6d0b7fe640 1 -- 172.21.15.84:0/36547569 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cfc003520 con 0x7f6d040389e0 2024-03-29T10:36:44.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.479+0000 7f6d0b7fe640 1 -- 172.21.15.84:0/36547569 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 652270566 0 0) 0x7f6d040c54c0 con 0x7f6d04030660 2024-03-29T10:36:44.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.479+0000 7f6d0b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:44.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.479+0000 7f6d0b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:44.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.479+0000 7f6d0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:44.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.479+0000 7f6d0b7fe640 10 monclient(hunting): my global_id is 4296 2024-03-29T10:36:44.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.479+0000 7f6d0b7fe640 1 -- 172.21.15.84:0/36547569 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d04039840 con 0x7f6d04030660 2024-03-29T10:36:44.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.481+0000 7f6d0b7fe640 1 -- 172.21.15.84:0/36547569 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2380604619 0 0) 0x7f6d04039840 con 0x7f6d04030660 2024-03-29T10:36:44.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.481+0000 7f6d0b7fe640 1 -- 172.21.15.84:0/36547569 >> v1:172.21.15.84:6790/0 conn(0x7f6d040389e0 legacy=0x7f6d04033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:44.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.481+0000 7f6d0b7fe640 1 -- 172.21.15.84:0/36547569 >> v1:172.21.15.86:6789/0 conn(0x7f6d04038dc0 legacy=0x7f6d040cadd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:36:44.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.481+0000 7f6d0b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:36:44.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.481+0000 7f6d0b7fe640 1 monclient: found mon.noname-a 2024-03-29T10:36:44.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.481+0000 7f6d0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:44.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.481+0000 7f6d0b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T10:36:44.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.481+0000 7f6d0b7fe640 1 -- 172.21.15.84:0/36547569 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d04035690 con 0x7f6d04030660 2024-03-29T10:36:44.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.481+0000 7f6d0b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:36:44.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.481+0000 7f6d0b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:36:44.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.481+0000 7f6d0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:44.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.481+0000 7f6d0b7fe640 1 -- 172.21.15.84:0/36547569 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f6d0c04f730 con 0x7f6d04030660 2024-03-29T10:36:44.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.481+0000 7f6d0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:44.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.481+0000 7f6d0b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:36:44.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.481+0000 7f6d0b7fe640 10 monclient: dump: 2024-03-29T10:36:44.470 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:44.470 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:44.470 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:44.470 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:44.470 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:44.471 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:44.471 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:44.471 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:44.471 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:44.471 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:44.471 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.481+0000 7f6d1192d640 5 monclient: authenticate success, global_id 4296 2024-03-29T10:36:44.471 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.481+0000 7f6d1192d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:44.471 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.481+0000 7f6d0b7fe640 1 -- 172.21.15.84:0/36547569 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d0c075f10 con 0x7f6d04030660 2024-03-29T10:36:44.471 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.481+0000 7f6d0b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:44.471 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.481+0000 7f6d0b7fe640 1 -- 172.21.15.84:0/36547569 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f6d0c0758b0 con 0x7f6d04030660 2024-03-29T10:36:44.471 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.481+0000 7f6d0b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:44.471 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.481+0000 7f6d0b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:36:44.471 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.481+0000 7f6d0b7fe640 10 monclient: dump: 2024-03-29T10:36:44.471 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:44.471 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:44.472 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:44.472 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:44.472 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:44.472 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:44.472 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:44.472 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:44.472 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:44.472 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:44.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.481+0000 7f6d1192d640 10 monclient: get_monmap_and_config success 2024-03-29T10:36:44.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.481+0000 7f6d1192d640 4 set_mon_vals no callback set 2024-03-29T10:36:44.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.482+0000 7f6d1192d640 10 monclient: shutdown 2024-03-29T10:36:44.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.482+0000 7f6d1192d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:44.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.482+0000 7f6d1192d640 1 -- 172.21.15.84:0/36547569 >> v1:172.21.15.84:6789/0 conn(0x7f6d04030660 legacy=0x7f6d040d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:44.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.482+0000 7f6d1192d640 1 -- 172.21.15.84:0/36547569 shutdown_connections 2024-03-29T10:36:44.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.482+0000 7f6d1192d640 1 -- 172.21.15.84:0/36547569 >> 172.21.15.84:0/36547569 conn(0x7f6d04005c00 msgr2=0x7f6d04006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:44.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.483+0000 7f6d1192d640 1 -- 172.21.15.84:0/36547569 shutdown_connections 2024-03-29T10:36:44.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.483+0000 7f6d1192d640 1 -- 172.21.15.84:0/36547569 wait complete. 2024-03-29T10:36:44.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.484+0000 7f6d1192d640 10 monclient: build_initial_monmap 2024-03-29T10:36:44.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.484+0000 7f6d1192d640 10 monclient: monmap: 2024-03-29T10:36:44.474 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:36:44.474 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:36:44.474 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:36:44.474 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:36:44.475 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:44.475 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:44.475 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:36:44.475 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:36:44.475 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:36:44.475 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:44.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.484+0000 7f6d1192d640 1 librados: starting msgr at 2024-03-29T10:36:44.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.484+0000 7f6d1192d640 1 librados: starting objecter 2024-03-29T10:36:44.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.484+0000 7f6d1192d640 1 Processor -- start 2024-03-29T10:36:44.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.485+0000 7f6d1192d640 1 -- start start 2024-03-29T10:36:44.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.485+0000 7f6d1192d640 1 librados: setting wanted keys 2024-03-29T10:36:44.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.485+0000 7f6d1192d640 1 librados: calling monclient init 2024-03-29T10:36:44.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.485+0000 7f6d1192d640 10 monclient: init 2024-03-29T10:36:44.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.486+0000 7f6d1192d640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:44.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.486+0000 7f6d1192d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:36:44.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.486+0000 7f6d1192d640 10 monclient(hunting): picked mon.noname-c con 0x7f6d04030660 addr v1:172.21.15.84:6790/0 2024-03-29T10:36:44.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.486+0000 7f6d1192d640 10 monclient(hunting): picked mon.noname-a con 0x7f6d040389e0 addr v1:172.21.15.84:6789/0 2024-03-29T10:36:44.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.488+0000 7f6d1192d640 10 monclient(hunting): picked mon.noname-b con 0x7f6d0402f690 addr v1:172.21.15.86:6789/0 2024-03-29T10:36:44.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.488+0000 7f6d1192d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d040c72f0 con 0x7f6d040389e0 2024-03-29T10:36:44.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.488+0000 7f6d1192d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d0400a230 con 0x7f6d0402f690 2024-03-29T10:36:44.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.488+0000 7f6d1192d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d0400a510 con 0x7f6d04030660 2024-03-29T10:36:44.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.488+0000 7f6d1192d640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:44.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.488+0000 7f6d1192d640 10 monclient(hunting): authenticate will time out at 906.752486s 2024-03-29T10:36:44.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.492+0000 7f6d1112c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f6d0402f690 0x7f6d0416bfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:36669/0 (socket says 172.21.15.84:36669) 2024-03-29T10:36:44.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.492+0000 7f6d1112c640 1 -- 172.21.15.84:0/2348567556 learned_addr learned my addr 172.21.15.84:0/2348567556 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:36:44.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.492+0000 7f6d097fa640 10 client.?.objecter ms_handle_connect 0x7f6d0402f690 2024-03-29T10:36:44.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.492+0000 7f6d097fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:44.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.492+0000 7f6d097fa640 1 -- 172.21.15.84:0/2348567556 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1750844573 0 0) 0x7f6d0400a230 con 0x7f6d0402f690 2024-03-29T10:36:44.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.492+0000 7f6d097fa640 10 monclient(hunting): _negotiate 2024-03-29T10:36:44.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.492+0000 7f6d097fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:44.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.492+0000 7f6d097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:44.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.492+0000 7f6d097fa640 10 monclient(hunting): my global_id is 4291 2024-03-29T10:36:44.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.492+0000 7f6d097fa640 1 -- 172.21.15.84:0/2348567556 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cf0004eb0 con 0x7f6d0402f690 2024-03-29T10:36:44.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.494+0000 7f6d097fa640 10 client.?.objecter ms_handle_connect 0x7f6d04030660 2024-03-29T10:36:44.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.494+0000 7f6d097fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:44.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.494+0000 7f6d097fa640 10 client.?.objecter ms_handle_connect 0x7f6d040389e0 2024-03-29T10:36:44.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.494+0000 7f6d097fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:44.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.494+0000 7f6d097fa640 1 -- 172.21.15.84:0/2348567556 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2122035211 0 0) 0x7f6d0400a510 con 0x7f6d04030660 2024-03-29T10:36:44.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.494+0000 7f6d097fa640 10 monclient(hunting): _negotiate 2024-03-29T10:36:44.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.494+0000 7f6d097fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:44.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.494+0000 7f6d097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:44.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.494+0000 7f6d097fa640 10 monclient(hunting): my global_id is 4310 2024-03-29T10:36:44.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.494+0000 7f6d097fa640 1 -- 172.21.15.84:0/2348567556 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d0400a230 con 0x7f6d04030660 2024-03-29T10:36:44.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.494+0000 7f6d097fa640 1 -- 172.21.15.84:0/2348567556 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1589711472 0 0) 0x7f6cf0004eb0 con 0x7f6d0402f690 2024-03-29T10:36:44.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.494+0000 7f6d097fa640 1 -- 172.21.15.84:0/2348567556 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d0400a510 con 0x7f6d0402f690 2024-03-29T10:36:44.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.494+0000 7f6d097fa640 1 -- 172.21.15.84:0/2348567556 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f6d0c04f400 con 0x7f6d0402f690 2024-03-29T10:36:44.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.494+0000 7f6d097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:44.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.494+0000 7f6d097fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:36:44.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.494+0000 7f6d097fa640 10 monclient(hunting): dump: 2024-03-29T10:36:44.486 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:44.486 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:44.486 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:44.486 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:44.486 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:44.486 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:44.486 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:44.486 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:44.486 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:44.486 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:44.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.494+0000 7f6d097fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:36:44.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.494+0000 7f6d097fa640 1 -- 172.21.15.84:0/2348567556 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851457845 0 0) 0x7f6d040c72f0 con 0x7f6d040389e0 2024-03-29T10:36:44.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.495+0000 7f6d097fa640 10 monclient(hunting): _negotiate 2024-03-29T10:36:44.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.495+0000 7f6d097fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:44.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.495+0000 7f6d097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:44.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.495+0000 7f6d097fa640 10 monclient(hunting): my global_id is 4299 2024-03-29T10:36:44.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.495+0000 7f6d097fa640 1 -- 172.21.15.84:0/2348567556 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cf0004eb0 con 0x7f6d040389e0 2024-03-29T10:36:44.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.495+0000 7f6d097fa640 1 -- 172.21.15.84:0/2348567556 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4092833862 0 0) 0x7f6d0400a510 con 0x7f6d0402f690 2024-03-29T10:36:44.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.495+0000 7f6d097fa640 1 -- 172.21.15.84:0/2348567556 >> v1:172.21.15.84:6790/0 conn(0x7f6d04030660 legacy=0x7f6d040c7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:44.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.495+0000 7f6d097fa640 1 -- 172.21.15.84:0/2348567556 >> v1:172.21.15.84:6789/0 conn(0x7f6d040389e0 legacy=0x7f6d040c7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:44.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.495+0000 7f6d097fa640 10 monclient: _finish_hunting 0 2024-03-29T10:36:44.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.495+0000 7f6d097fa640 1 monclient: found mon.b 2024-03-29T10:36:44.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.495+0000 7f6d097fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:44.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.495+0000 7f6d097fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:36:44.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.495+0000 7f6d097fa640 1 -- 172.21.15.84:0/2348567556 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d040c6660 con 0x7f6d0402f690 2024-03-29T10:36:44.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.495+0000 7f6d097fa640 10 monclient: _finish_auth 0 2024-03-29T10:36:44.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.495+0000 7f6d097fa640 10 monclient: _check_auth_tickets 2024-03-29T10:36:44.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.495+0000 7f6d097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:44.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.495+0000 7f6d1192d640 5 monclient: authenticate success, global_id 4291 2024-03-29T10:36:44.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.495+0000 7f6d1192d640 10 monclient: _renew_subs 2024-03-29T10:36:44.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.495+0000 7f6d1192d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:36:44.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.495+0000 7f6d1192d640 1 -- 172.21.15.84:0/2348567556 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d040c2cf0 con 0x7f6d0402f690 2024-03-29T10:36:44.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.495+0000 7f6d1192d640 10 client.4291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:36:44.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.495+0000 7f6d1192d640 10 monclient: _renew_subs 2024-03-29T10:36:44.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.495+0000 7f6d1192d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:36:44.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.495+0000 7f6d1192d640 1 -- 172.21.15.84:0/2348567556 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d0416d8c0 con 0x7f6d0402f690 2024-03-29T10:36:44.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.495+0000 7f6d1192d640 1 librados: init done 2024-03-29T10:36:44.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.497+0000 7f6cf6ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:44.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.497+0000 7f6cf6ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:44.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.497+0000 7f6cf6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:36:44.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.497+0000 7f6cf6ffd640 1 -- 172.21.15.84:0/2348567556 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d0400aa30 con 0x7f6d0402f690 2024-03-29T10:36:44.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.499+0000 7f6d097fa640 1 -- 172.21.15.84:0/2348567556 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d0c04fdd0 con 0x7f6d0402f690 2024-03-29T10:36:44.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.499+0000 7f6d097fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:44.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.499+0000 7f6d0affd640 4 set_mon_vals no callback set 2024-03-29T10:36:44.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.501+0000 7f6d097fa640 1 -- 172.21.15.84:0/2348567556 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f6d0c04f730 con 0x7f6d0402f690 2024-03-29T10:36:44.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.501+0000 7f6d097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:44.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.501+0000 7f6d097fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:36:44.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.501+0000 7f6d097fa640 10 monclient: dump: 2024-03-29T10:36:44.490 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:44.490 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:44.490 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:44.490 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:44.490 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:44.490 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:44.490 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:44.490 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:44.490 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:44.490 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:44.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.502+0000 7f6d097fa640 1 -- 172.21.15.84:0/2348567556 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f6d0c067bf0 con 0x7f6d0402f690 2024-03-29T10:36:44.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.503+0000 7f6d097fa640 1 -- 172.21.15.84:0/2348567556 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f6d0c0ff210 con 0x7f6d0402f690 2024-03-29T10:36:44.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.503+0000 7f6d097fa640 10 client.4291.objecter ms_dispatch 0x7f6d0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:36:44.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.503+0000 7f6d097fa640 3 client.4291.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:36:44.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.503+0000 7f6d097fa640 3 client.4291.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:36:44.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.503+0000 7f6d097fa640 20 client.4291.objecter dump_active .. 0 homeless 2024-03-29T10:36:44.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.507+0000 7f6d1092b640 10 monclient: get_auth_request con 0x7f6cf006dc30 auth_method 0 2024-03-29T10:36:44.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.507+0000 7f6d097fa640 1 -- 172.21.15.84:0/2348567556 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6d0c0caaf0 con 0x7f6d0402f690 2024-03-29T10:36:44.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.507+0000 7f6d097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:44.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.507+0000 7f6d097fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:36:44.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.507+0000 7f6d097fa640 10 client.4291.objecter ms_handle_connect 0x7f6cf006dc30 2024-03-29T10:36:44.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.827+0000 7f6cf6ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:44.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.827+0000 7f6cf6ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:44.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.827+0000 7f6cf6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:36:44.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.827+0000 7f6cf6ffd640 1 -- 172.21.15.84:0/2348567556 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d040c74a0 con 0x7f6d0402f690 2024-03-29T10:36:44.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.828+0000 7f6d097fa640 1 -- 172.21.15.84:0/2348567556 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f6d040c74a0 con 0x7f6d0402f690 2024-03-29T10:36:44.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.828+0000 7f6d097fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:44.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.828+0000 7f6d097fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:36:44.815 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:36:44.815 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:36:44.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.832+0000 7f6cf6ffd640 10 librados: watch_flush enter 2024-03-29T10:36:44.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.832+0000 7f6cf6ffd640 10 librados: watch_flush exit 2024-03-29T10:36:44.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.832+0000 7f6cf6ffd640 20 client.4291.objecter shutdown clearing up homeless session... 2024-03-29T10:36:44.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.832+0000 7f6cf6ffd640 10 client.4291.objecter successfully canceled tick 2024-03-29T10:36:44.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.833+0000 7f6cf6ffd640 1 -- 172.21.15.84:0/2348567556 >> v1:172.21.15.86:6816/3337225337 conn(0x7f6cf006dc30 legacy=0x7f6cf00700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:44.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.833+0000 7f6cf6ffd640 10 monclient: shutdown 2024-03-29T10:36:44.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.833+0000 7f6cf6ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:44.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.833+0000 7f6cf6ffd640 1 -- 172.21.15.84:0/2348567556 >> v1:172.21.15.86:6789/0 conn(0x7f6d0402f690 legacy=0x7f6d0416bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:44.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.834+0000 7f6cf6ffd640 1 -- 172.21.15.84:0/2348567556 shutdown_connections 2024-03-29T10:36:44.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.834+0000 7f6cf6ffd640 1 -- 172.21.15.84:0/2348567556 >> 172.21.15.84:0/2348567556 conn(0x7f6d04005c00 msgr2=0x7f6d040c43b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:44.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.836+0000 7f6cf6ffd640 1 -- 172.21.15.84:0/2348567556 shutdown_connections 2024-03-29T10:36:44.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.836+0000 7f6cf6ffd640 1 -- 172.21.15.84:0/2348567556 wait complete. 2024-03-29T10:36:44.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:44.836+0000 7f6cf6ffd640 1 librados: shutdown 2024-03-29T10:36:44.841 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:36:51.649 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:36:51.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.822+0000 7fa559455640 10 monclient: get_monmap_and_config 2024-03-29T10:36:51.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.822+0000 7fa559455640 10 monclient: build_initial_monmap 2024-03-29T10:36:51.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.822+0000 7fa559455640 10 monclient: monmap: 2024-03-29T10:36:51.808 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:36:51.808 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:51.808 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:36:51.823902+0000 2024-03-29T10:36:51.808 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:36:51.823902+0000 2024-03-29T10:36:51.809 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:51.809 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:51.809 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:36:51.809 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:36:51.809 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:36:51.809 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:51.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.824+0000 7fa559455640 1 Processor -- start 2024-03-29T10:36:51.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.824+0000 7fa559455640 1 -- start start 2024-03-29T10:36:51.809 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.824+0000 7fa559455640 10 monclient: init 2024-03-29T10:36:51.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.825+0000 7fa559455640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:51.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.825+0000 7fa559455640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:36:51.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.825+0000 7fa559455640 10 monclient(hunting): picked mon.noname-b con 0x7fa54c006e60 addr v1:172.21.15.84:6790/0 2024-03-29T10:36:51.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.825+0000 7fa559455640 10 monclient(hunting): picked mon.noname-a con 0x7fa54c035e30 addr v1:172.21.15.84:6789/0 2024-03-29T10:36:51.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.825+0000 7fa559455640 10 monclient(hunting): picked mon.noname-c con 0x7fa54c0c96a0 addr v1:172.21.15.86:6789/0 2024-03-29T10:36:51.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.825+0000 7fa559455640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa54c0359a0 con 0x7fa54c035e30 2024-03-29T10:36:51.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.825+0000 7fa559455640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa54c0362b0 con 0x7fa54c0c96a0 2024-03-29T10:36:51.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.825+0000 7fa559455640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa54c035670 con 0x7fa54c006e60 2024-03-29T10:36:51.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.825+0000 7fa559455640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:51.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.825+0000 7fa559455640 10 monclient(hunting): authenticate will time out at 914.088486s 2024-03-29T10:36:51.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.828+0000 7fa552ffd640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fa54c006e60 0x7fa54c0cbcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:46077/0 (socket says 172.21.15.84:46077) 2024-03-29T10:36:51.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.828+0000 7fa552ffd640 1 -- 172.21.15.84:0/405668126 learned_addr learned my addr 172.21.15.84:0/405668126 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:36:51.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.829+0000 7fa551ffb640 1 -- 172.21.15.84:0/405668126 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656898897 0 0) 0x7fa54c035670 con 0x7fa54c006e60 2024-03-29T10:36:51.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.829+0000 7fa551ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:36:51.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.829+0000 7fa551ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:51.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.829+0000 7fa551ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:51.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.829+0000 7fa551ffb640 10 monclient(hunting): my global_id is 4313 2024-03-29T10:36:51.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.829+0000 7fa551ffb640 1 -- 172.21.15.84:0/405668126 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa53c003520 con 0x7fa54c006e60 2024-03-29T10:36:51.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.830+0000 7fa551ffb640 1 -- 172.21.15.84:0/405668126 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1425792790 0 0) 0x7fa54c0359a0 con 0x7fa54c035e30 2024-03-29T10:36:51.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.830+0000 7fa551ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:36:51.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.830+0000 7fa551ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:51.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.830+0000 7fa551ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:51.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.830+0000 7fa551ffb640 10 monclient(hunting): my global_id is 4302 2024-03-29T10:36:51.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.830+0000 7fa551ffb640 1 -- 172.21.15.84:0/405668126 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa54c035670 con 0x7fa54c035e30 2024-03-29T10:36:51.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.830+0000 7fa551ffb640 1 -- 172.21.15.84:0/405668126 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 737545760 0 0) 0x7fa54c035670 con 0x7fa54c035e30 2024-03-29T10:36:51.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.830+0000 7fa551ffb640 1 -- 172.21.15.84:0/405668126 >> v1:172.21.15.84:6790/0 conn(0x7fa54c006e60 legacy=0x7fa54c0cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:51.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.830+0000 7fa551ffb640 1 -- 172.21.15.84:0/405668126 >> v1:172.21.15.86:6789/0 conn(0x7fa54c0c96a0 legacy=0x7fa54c0d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:51.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.830+0000 7fa551ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:36:51.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.830+0000 7fa551ffb640 1 monclient: found mon.noname-a 2024-03-29T10:36:51.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.830+0000 7fa551ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:51.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.830+0000 7fa551ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T10:36:51.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.830+0000 7fa551ffb640 1 -- 172.21.15.84:0/405668126 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54c034e50 con 0x7fa54c035e30 2024-03-29T10:36:51.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.830+0000 7fa551ffb640 10 monclient: _finish_auth 0 2024-03-29T10:36:51.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.830+0000 7fa551ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:36:51.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.830+0000 7fa551ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:51.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.830+0000 7fa551ffb640 1 -- 172.21.15.84:0/405668126 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa544001670 con 0x7fa54c035e30 2024-03-29T10:36:51.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.830+0000 7fa551ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:51.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.830+0000 7fa551ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:36:51.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.830+0000 7fa551ffb640 10 monclient: dump: 2024-03-29T10:36:51.818 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:51.818 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:51.818 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:51.818 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:51.818 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:51.818 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:51.818 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:51.818 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:51.818 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:51.818 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:51.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.830+0000 7fa559455640 5 monclient: authenticate success, global_id 4302 2024-03-29T10:36:51.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.830+0000 7fa559455640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:51.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.831+0000 7fa551ffb640 1 -- 172.21.15.84:0/405668126 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa544001900 con 0x7fa54c035e30 2024-03-29T10:36:51.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.831+0000 7fa551ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:51.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.831+0000 7fa559455640 10 monclient: get_monmap_and_config success 2024-03-29T10:36:51.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.831+0000 7fa559455640 4 set_mon_vals no callback set 2024-03-29T10:36:51.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.831+0000 7fa559455640 10 monclient: shutdown 2024-03-29T10:36:51.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.831+0000 7fa559455640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:51.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.831+0000 7fa559455640 1 -- 172.21.15.84:0/405668126 >> v1:172.21.15.84:6789/0 conn(0x7fa54c035e30 legacy=0x7fa54c0c8f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:51.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.831+0000 7fa551ffb640 1 -- 172.21.15.84:0/405668126 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa544001670 con 0x7fa54c035e30 2024-03-29T10:36:51.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.832+0000 7fa551ffb640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:36:51.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.832+0000 7fa559455640 1 -- 172.21.15.84:0/405668126 shutdown_connections 2024-03-29T10:36:51.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.832+0000 7fa559455640 1 -- 172.21.15.84:0/405668126 >> 172.21.15.84:0/405668126 conn(0x7fa54c005c30 msgr2=0x7fa54c006040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:51.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.833+0000 7fa559455640 1 -- 172.21.15.84:0/405668126 shutdown_connections 2024-03-29T10:36:51.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.834+0000 7fa559455640 1 -- 172.21.15.84:0/405668126 wait complete. 2024-03-29T10:36:51.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.834+0000 7fa559455640 10 monclient: build_initial_monmap 2024-03-29T10:36:51.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.834+0000 7fa559455640 10 monclient: monmap: 2024-03-29T10:36:51.822 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:36:51.822 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:36:51.822 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:36:51.822 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:36:51.822 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:51.822 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:51.822 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:36:51.822 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:36:51.822 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:36:51.822 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:51.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.835+0000 7fa559455640 1 librados: starting msgr at 2024-03-29T10:36:51.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.835+0000 7fa559455640 1 librados: starting objecter 2024-03-29T10:36:51.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.835+0000 7fa559455640 1 Processor -- start 2024-03-29T10:36:51.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.835+0000 7fa559455640 1 -- start start 2024-03-29T10:36:51.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.835+0000 7fa559455640 1 librados: setting wanted keys 2024-03-29T10:36:51.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.835+0000 7fa559455640 1 librados: calling monclient init 2024-03-29T10:36:51.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.835+0000 7fa559455640 10 monclient: init 2024-03-29T10:36:51.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.836+0000 7fa559455640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:51.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.836+0000 7fa559455640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:36:51.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.836+0000 7fa559455640 10 monclient(hunting): picked mon.noname-b con 0x7fa54c006e60 addr v1:172.21.15.86:6789/0 2024-03-29T10:36:51.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.839+0000 7fa559455640 10 monclient(hunting): picked mon.noname-c con 0x7fa54c035e30 addr v1:172.21.15.84:6790/0 2024-03-29T10:36:51.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.839+0000 7fa559455640 10 monclient(hunting): picked mon.noname-a con 0x7fa54c0c96a0 addr v1:172.21.15.84:6789/0 2024-03-29T10:36:51.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.839+0000 7fa559455640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa54c0cde30 con 0x7fa54c0c96a0 2024-03-29T10:36:51.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.839+0000 7fa559455640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa54c02ebb0 con 0x7fa54c006e60 2024-03-29T10:36:51.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.839+0000 7fa559455640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa54c02ed40 con 0x7fa54c035e30 2024-03-29T10:36:51.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.839+0000 7fa559455640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:51.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.839+0000 7fa559455640 10 monclient(hunting): authenticate will time out at 914.102931s 2024-03-29T10:36:51.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.840+0000 7fa552ffd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa54c006e60 0x7fa54c033730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:56393/0 (socket says 172.21.15.84:56393) 2024-03-29T10:36:51.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.840+0000 7fa552ffd640 1 -- 172.21.15.84:0/536951318 learned_addr learned my addr 172.21.15.84:0/536951318 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:36:51.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.844+0000 7fa53b7fe640 10 client.?.objecter ms_handle_connect 0x7fa54c006e60 2024-03-29T10:36:51.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.844+0000 7fa53b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:51.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.844+0000 7fa53b7fe640 1 -- 172.21.15.84:0/536951318 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2962217534 0 0) 0x7fa54c02ebb0 con 0x7fa54c006e60 2024-03-29T10:36:51.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.844+0000 7fa53b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:51.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.844+0000 7fa53b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:51.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.844+0000 7fa53b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:51.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.844+0000 7fa53b7fe640 10 monclient(hunting): my global_id is 4297 2024-03-29T10:36:51.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.844+0000 7fa53b7fe640 1 -- 172.21.15.84:0/536951318 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa540004eb0 con 0x7fa54c006e60 2024-03-29T10:36:51.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.848+0000 7fa53b7fe640 10 client.?.objecter ms_handle_connect 0x7fa54c0c96a0 2024-03-29T10:36:51.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.848+0000 7fa53b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:51.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.848+0000 7fa53b7fe640 10 client.?.objecter ms_handle_connect 0x7fa54c035e30 2024-03-29T10:36:51.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.848+0000 7fa53b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:51.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.848+0000 7fa53b7fe640 1 -- 172.21.15.84:0/536951318 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3651802037 0 0) 0x7fa540004eb0 con 0x7fa54c006e60 2024-03-29T10:36:51.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.848+0000 7fa53b7fe640 1 -- 172.21.15.84:0/536951318 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa54c02ebb0 con 0x7fa54c006e60 2024-03-29T10:36:51.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.848+0000 7fa53b7fe640 1 -- 172.21.15.84:0/536951318 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3060221512 0 0) 0x7fa54c02ed40 con 0x7fa54c035e30 2024-03-29T10:36:51.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.848+0000 7fa53b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:36:51.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.848+0000 7fa53b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:51.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.848+0000 7fa53b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:51.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.848+0000 7fa53b7fe640 10 monclient(hunting): my global_id is 4316 2024-03-29T10:36:51.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.848+0000 7fa53b7fe640 1 -- 172.21.15.84:0/536951318 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa540004eb0 con 0x7fa54c035e30 2024-03-29T10:36:51.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.848+0000 7fa53b7fe640 1 -- 172.21.15.84:0/536951318 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa55404fdd0 con 0x7fa54c006e60 2024-03-29T10:36:51.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.848+0000 7fa53b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:51.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.848+0000 7fa53b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:36:51.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.848+0000 7fa53b7fe640 10 monclient(hunting): dump: 2024-03-29T10:36:51.837 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:51.837 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:51.837 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:51.837 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:51.837 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:51.837 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:51.837 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:51.837 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:51.837 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:51.837 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:51.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.848+0000 7fa53b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:36:51.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.849+0000 7fa53b7fe640 1 -- 172.21.15.84:0/536951318 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1294181031 0 0) 0x7fa540004eb0 con 0x7fa54c035e30 2024-03-29T10:36:51.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.849+0000 7fa53b7fe640 1 -- 172.21.15.84:0/536951318 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa54c02ed40 con 0x7fa54c035e30 2024-03-29T10:36:51.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.849+0000 7fa53b7fe640 1 -- 172.21.15.84:0/536951318 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa5440035f0 con 0x7fa54c035e30 2024-03-29T10:36:51.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.849+0000 7fa53b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:51.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.849+0000 7fa53b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:51.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.849+0000 7fa53b7fe640 10 monclient(hunting): dump: 2024-03-29T10:36:51.838 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:51.838 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:51.838 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:51.838 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:51.838 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:51.838 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:51.839 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:51.839 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:51.839 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:51.839 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:51.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.849+0000 7fa53b7fe640 1 -- 172.21.15.84:0/536951318 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1582117763 0 0) 0x7fa54c02ed40 con 0x7fa54c035e30 2024-03-29T10:36:51.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.849+0000 7fa53b7fe640 1 -- 172.21.15.84:0/536951318 >> v1:172.21.15.86:6789/0 conn(0x7fa54c006e60 legacy=0x7fa54c033730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:51.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.849+0000 7fa53b7fe640 1 -- 172.21.15.84:0/536951318 >> v1:172.21.15.84:6789/0 conn(0x7fa54c0c96a0 legacy=0x7fa54c185a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:51.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.849+0000 7fa53b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:36:51.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.849+0000 7fa53b7fe640 1 monclient: found mon.c 2024-03-29T10:36:51.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.849+0000 7fa53b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:51.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.849+0000 7fa53b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:36:51.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.849+0000 7fa53b7fe640 1 -- 172.21.15.84:0/536951318 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54c0c51b0 con 0x7fa54c035e30 2024-03-29T10:36:51.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.849+0000 7fa53b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:36:51.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.849+0000 7fa53b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:36:51.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.849+0000 7fa53b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:51.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.849+0000 7fa559455640 5 monclient: authenticate success, global_id 4316 2024-03-29T10:36:51.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.849+0000 7fa559455640 10 monclient: _renew_subs 2024-03-29T10:36:51.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.849+0000 7fa559455640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:36:51.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.849+0000 7fa559455640 1 -- 172.21.15.84:0/536951318 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa54c0c5460 con 0x7fa54c035e30 2024-03-29T10:36:51.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.849+0000 7fa559455640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:36:51.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.849+0000 7fa559455640 10 monclient: _renew_subs 2024-03-29T10:36:51.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.849+0000 7fa559455640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:36:51.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.849+0000 7fa559455640 1 -- 172.21.15.84:0/536951318 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa54c02ff70 con 0x7fa54c035e30 2024-03-29T10:36:51.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.849+0000 7fa559455640 1 librados: init done 2024-03-29T10:36:51.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.851+0000 7fa53b7fe640 1 -- 172.21.15.84:0/536951318 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa544001250 con 0x7fa54c035e30 2024-03-29T10:36:51.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.851+0000 7fa53b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:51.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.851+0000 7fa5517fa640 4 set_mon_vals no callback set 2024-03-29T10:36:51.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.852+0000 7fa559455640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:51.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.852+0000 7fa53b7fe640 1 -- 172.21.15.84:0/536951318 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa544003ba0 con 0x7fa54c035e30 2024-03-29T10:36:51.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.852+0000 7fa53b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:51.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.852+0000 7fa53b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:51.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.852+0000 7fa53b7fe640 10 monclient: dump: 2024-03-29T10:36:51.843 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:51.843 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:51.843 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:51.843 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:51.843 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:51.843 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:51.843 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:51.844 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:51.844 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:51.844 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:51.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.852+0000 7fa559455640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:51.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.852+0000 7fa559455640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:36:51.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.852+0000 7fa559455640 1 -- 172.21.15.84:0/536951318 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa534005080 con 0x7fa54c035e30 2024-03-29T10:36:51.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.853+0000 7fa53b7fe640 1 -- 172.21.15.84:0/536951318 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7fa544019b70 con 0x7fa54c035e30 2024-03-29T10:36:51.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.853+0000 7fa53b7fe640 1 -- 172.21.15.84:0/536951318 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7fa544083900 con 0x7fa54c035e30 2024-03-29T10:36:51.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.853+0000 7fa53b7fe640 10 client.4316.objecter ms_dispatch 0x7fa554000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:36:51.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.853+0000 7fa53b7fe640 3 client.4316.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:36:51.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.853+0000 7fa53b7fe640 3 client.4316.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:36:51.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.854+0000 7fa53b7fe640 20 client.4316.objecter dump_active .. 0 homeless 2024-03-29T10:36:51.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.856+0000 7fa552ffd640 10 monclient: get_auth_request con 0x7fa54006dd50 auth_method 0 2024-03-29T10:36:51.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.858+0000 7fa53b7fe640 10 client.4316.objecter ms_handle_connect 0x7fa54006dd50 2024-03-29T10:36:51.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.860+0000 7fa53b7fe640 1 -- 172.21.15.84:0/536951318 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa54404f300 con 0x7fa54c035e30 2024-03-29T10:36:51.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.860+0000 7fa53b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:51.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:51.860+0000 7fa53b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:36:52.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:52.176+0000 7fa559455640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:52.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:52.176+0000 7fa559455640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:52.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:52.176+0000 7fa559455640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:36:52.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:52.176+0000 7fa559455640 1 -- 172.21.15.84:0/536951318 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa534005330 con 0x7fa54c035e30 2024-03-29T10:36:52.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:52.176+0000 7fa53b7fe640 1 -- 172.21.15.84:0/536951318 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7fa544051230 con 0x7fa54c035e30 2024-03-29T10:36:52.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:52.176+0000 7fa53b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:52.162 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:52.176+0000 7fa53b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:36:52.162 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:36:52.162 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:36:52.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:52.180+0000 7fa559455640 10 librados: watch_flush enter 2024-03-29T10:36:52.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:52.180+0000 7fa559455640 10 librados: watch_flush exit 2024-03-29T10:36:52.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:52.180+0000 7fa559455640 20 client.4316.objecter shutdown clearing up homeless session... 2024-03-29T10:36:52.165 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:52.180+0000 7fa559455640 10 client.4316.objecter successfully canceled tick 2024-03-29T10:36:52.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:52.180+0000 7fa559455640 1 -- 172.21.15.84:0/536951318 >> v1:172.21.15.86:6816/3337225337 conn(0x7fa54006dd50 legacy=0x7fa540070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:52.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:52.181+0000 7fa559455640 10 monclient: shutdown 2024-03-29T10:36:52.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:52.181+0000 7fa559455640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:52.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:52.181+0000 7fa559455640 1 -- 172.21.15.84:0/536951318 >> v1:172.21.15.84:6790/0 conn(0x7fa54c035e30 legacy=0x7fa54c02f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:52.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:52.181+0000 7fa559455640 1 -- 172.21.15.84:0/536951318 shutdown_connections 2024-03-29T10:36:52.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:52.181+0000 7fa559455640 1 -- 172.21.15.84:0/536951318 >> 172.21.15.84:0/536951318 conn(0x7fa54c005c30 msgr2=0x7fa54c0c8bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:52.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:52.181+0000 7fa559455640 1 -- 172.21.15.84:0/536951318 shutdown_connections 2024-03-29T10:36:52.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:52.181+0000 7fa559455640 1 -- 172.21.15.84:0/536951318 wait complete. 2024-03-29T10:36:52.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:52.181+0000 7fa559455640 1 librados: shutdown 2024-03-29T10:36:52.183 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:36:58.990 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:36:59.139 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.154+0000 7fe19581e640 10 monclient: get_monmap_and_config 2024-03-29T10:36:59.139 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.154+0000 7fe19581e640 10 monclient: build_initial_monmap 2024-03-29T10:36:59.139 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.154+0000 7fe19581e640 10 monclient: monmap: 2024-03-29T10:36:59.140 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:36:59.140 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:59.140 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:36:59.155866+0000 2024-03-29T10:36:59.140 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:36:59.155866+0000 2024-03-29T10:36:59.140 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:59.140 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:59.140 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:36:59.140 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:36:59.140 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:36:59.140 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:59.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.156+0000 7fe19581e640 1 Processor -- start 2024-03-29T10:36:59.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.156+0000 7fe19581e640 1 -- start start 2024-03-29T10:36:59.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.156+0000 7fe19581e640 10 monclient: init 2024-03-29T10:36:59.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.156+0000 7fe19581e640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:59.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.156+0000 7fe19581e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:36:59.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.156+0000 7fe19581e640 10 monclient(hunting): picked mon.noname-b con 0x7fe19006ade0 addr v1:172.21.15.84:6790/0 2024-03-29T10:36:59.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.156+0000 7fe19581e640 10 monclient(hunting): picked mon.noname-a con 0x7fe19006ed60 addr v1:172.21.15.84:6789/0 2024-03-29T10:36:59.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.156+0000 7fe19581e640 10 monclient(hunting): picked mon.noname-c con 0x7fe190072340 addr v1:172.21.15.86:6789/0 2024-03-29T10:36:59.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.156+0000 7fe19581e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe19006ff50 con 0x7fe19006ed60 2024-03-29T10:36:59.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.156+0000 7fe19581e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe190070180 con 0x7fe190072340 2024-03-29T10:36:59.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.156+0000 7fe19581e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe19006f310 con 0x7fe19006ade0 2024-03-29T10:36:59.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.156+0000 7fe19581e640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:59.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.157+0000 7fe19581e640 10 monclient(hunting): authenticate will time out at 921.420130s 2024-03-29T10:36:59.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.160+0000 7fe18effd640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fe19006ade0 0x7fe1901499e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:34413/0 (socket says 172.21.15.84:34413) 2024-03-29T10:36:59.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.160+0000 7fe18effd640 1 -- 172.21.15.84:0/507993398 learned_addr learned my addr 172.21.15.84:0/507993398 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:36:59.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.160+0000 7fe18dffb640 1 -- 172.21.15.84:0/507993398 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1998873848 0 0) 0x7fe19006f310 con 0x7fe19006ade0 2024-03-29T10:36:59.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.160+0000 7fe18dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:36:59.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.160+0000 7fe18dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:59.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.160+0000 7fe18dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:59.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.160+0000 7fe18dffb640 10 monclient(hunting): my global_id is 4319 2024-03-29T10:36:59.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.160+0000 7fe18dffb640 1 -- 172.21.15.84:0/507993398 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe178003520 con 0x7fe19006ade0 2024-03-29T10:36:59.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.160+0000 7fe18dffb640 1 -- 172.21.15.84:0/507993398 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3623993042 0 0) 0x7fe178003520 con 0x7fe19006ade0 2024-03-29T10:36:59.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.160+0000 7fe18dffb640 1 -- 172.21.15.84:0/507993398 >> v1:172.21.15.86:6789/0 conn(0x7fe190072340 legacy=0x7fe190151d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:36:59.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.160+0000 7fe18dffb640 1 -- 172.21.15.84:0/507993398 >> v1:172.21.15.84:6789/0 conn(0x7fe19006ed60 legacy=0x7fe190071c30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:36:59.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.163+0000 7fe18dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:36:59.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.163+0000 7fe18dffb640 1 monclient: found mon.noname-b 2024-03-29T10:36:59.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.163+0000 7fe18dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:59.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.163+0000 7fe18dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:36:59.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.163+0000 7fe18dffb640 1 -- 172.21.15.84:0/507993398 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1900aeed0 con 0x7fe19006ade0 2024-03-29T10:36:59.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.163+0000 7fe18dffb640 10 monclient: _finish_auth 0 2024-03-29T10:36:59.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.163+0000 7fe18dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:36:59.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.163+0000 7fe18dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:59.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.163+0000 7fe18dffb640 1 -- 172.21.15.84:0/507993398 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fe180001670 con 0x7fe19006ade0 2024-03-29T10:36:59.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.163+0000 7fe18dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:59.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.163+0000 7fe18dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:36:59.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.163+0000 7fe18dffb640 10 monclient: dump: 2024-03-29T10:36:59.149 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:59.149 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:59.149 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:59.149 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:59.149 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:59.149 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:59.149 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:59.149 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:59.149 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:59.149 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:59.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.164+0000 7fe19581e640 5 monclient: authenticate success, global_id 4319 2024-03-29T10:36:59.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.164+0000 7fe19581e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:36:59.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.167+0000 7fe18dffb640 1 -- 172.21.15.84:0/507993398 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1800018d0 con 0x7fe19006ade0 2024-03-29T10:36:59.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.167+0000 7fe18dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:59.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.168+0000 7fe18dffb640 1 -- 172.21.15.84:0/507993398 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fe180001d20 con 0x7fe19006ade0 2024-03-29T10:36:59.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.168+0000 7fe18dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:59.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.168+0000 7fe18dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:59.153 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.168+0000 7fe18dffb640 10 monclient: dump: 2024-03-29T10:36:59.153 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:59.153 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:59.153 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:59.153 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:59.153 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:59.153 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:59.153 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:59.153 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:59.153 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:59.153 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:59.154 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.168+0000 7fe19581e640 10 monclient: get_monmap_and_config success 2024-03-29T10:36:59.154 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.168+0000 7fe19581e640 4 set_mon_vals no callback set 2024-03-29T10:36:59.154 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.169+0000 7fe19581e640 10 monclient: shutdown 2024-03-29T10:36:59.154 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.169+0000 7fe19581e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:59.155 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.169+0000 7fe19581e640 1 -- 172.21.15.84:0/507993398 >> v1:172.21.15.84:6790/0 conn(0x7fe19006ade0 legacy=0x7fe1901499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:59.155 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.171+0000 7fe19581e640 1 -- 172.21.15.84:0/507993398 shutdown_connections 2024-03-29T10:36:59.155 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.171+0000 7fe19581e640 1 -- 172.21.15.84:0/507993398 >> 172.21.15.84:0/507993398 conn(0x7fe1900b4930 msgr2=0x7fe1900b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:59.156 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.172+0000 7fe19581e640 1 -- 172.21.15.84:0/507993398 shutdown_connections 2024-03-29T10:36:59.157 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.172+0000 7fe19581e640 1 -- 172.21.15.84:0/507993398 wait complete. 2024-03-29T10:36:59.158 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.173+0000 7fe19581e640 10 monclient: build_initial_monmap 2024-03-29T10:36:59.158 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.173+0000 7fe19581e640 10 monclient: monmap: 2024-03-29T10:36:59.158 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:36:59.158 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:36:59.158 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:36:59.158 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:36:59.158 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:36:59.158 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:59.158 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:36:59.158 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:36:59.158 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:36:59.158 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:59.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.173+0000 7fe19581e640 1 librados: starting msgr at 2024-03-29T10:36:59.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.173+0000 7fe19581e640 1 librados: starting objecter 2024-03-29T10:36:59.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.173+0000 7fe19581e640 1 Processor -- start 2024-03-29T10:36:59.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.173+0000 7fe19581e640 1 -- start start 2024-03-29T10:36:59.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.173+0000 7fe19581e640 1 librados: setting wanted keys 2024-03-29T10:36:59.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.173+0000 7fe19581e640 1 librados: calling monclient init 2024-03-29T10:36:59.159 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.173+0000 7fe19581e640 10 monclient: init 2024-03-29T10:36:59.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.174+0000 7fe19581e640 10 monclient: _reopen_session rank -1 2024-03-29T10:36:59.160 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.174+0000 7fe19581e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:36:59.161 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.177+0000 7fe19581e640 10 monclient(hunting): picked mon.noname-c con 0x7fe19006ade0 addr v1:172.21.15.84:6790/0 2024-03-29T10:36:59.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.177+0000 7fe19581e640 10 monclient(hunting): picked mon.noname-a con 0x7fe19006ed60 addr v1:172.21.15.84:6789/0 2024-03-29T10:36:59.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.181+0000 7fe18effd640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fe19006ade0 0x7fe1900a9780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:34515/0 (socket says 172.21.15.84:34515) 2024-03-29T10:36:59.166 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.181+0000 7fe18effd640 1 -- 172.21.15.84:0/2986533734 learned_addr learned my addr 172.21.15.84:0/2986533734 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:36:59.167 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.181+0000 7fe19581e640 10 monclient(hunting): picked mon.noname-b con 0x7fe190072340 addr v1:172.21.15.86:6789/0 2024-03-29T10:36:59.167 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.181+0000 7fe19581e640 1 -- 172.21.15.84:0/2986533734 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1900ae0b0 con 0x7fe19006ed60 2024-03-29T10:36:59.167 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.181+0000 7fe19581e640 1 -- 172.21.15.84:0/2986533734 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1900abfb0 con 0x7fe190072340 2024-03-29T10:36:59.167 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.181+0000 7fe19581e640 1 -- 172.21.15.84:0/2986533734 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1900ae300 con 0x7fe19006ade0 2024-03-29T10:36:59.167 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.182+0000 7fe19581e640 10 monclient(hunting): _renew_subs 2024-03-29T10:36:59.167 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.182+0000 7fe19581e640 10 monclient(hunting): authenticate will time out at 921.445210s 2024-03-29T10:36:59.167 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.182+0000 7fe177fff640 10 client.?.objecter ms_handle_connect 0x7fe19006ade0 2024-03-29T10:36:59.167 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.182+0000 7fe177fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:59.168 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.183+0000 7fe177fff640 1 -- 172.21.15.84:0/2986533734 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 227022417 0 0) 0x7fe1900ae300 con 0x7fe19006ade0 2024-03-29T10:36:59.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.183+0000 7fe177fff640 10 monclient(hunting): _negotiate 2024-03-29T10:36:59.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.184+0000 7fe177fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:59.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.184+0000 7fe177fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:59.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.184+0000 7fe177fff640 10 monclient(hunting): my global_id is 4322 2024-03-29T10:36:59.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.184+0000 7fe177fff640 1 -- 172.21.15.84:0/2986533734 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe17c004eb0 con 0x7fe19006ade0 2024-03-29T10:36:59.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.184+0000 7fe177fff640 10 client.?.objecter ms_handle_connect 0x7fe19006ed60 2024-03-29T10:36:59.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.184+0000 7fe177fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:36:59.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.184+0000 7fe177fff640 1 -- 172.21.15.84:0/2986533734 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850048161 0 0) 0x7fe1900ae0b0 con 0x7fe19006ed60 2024-03-29T10:36:59.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.184+0000 7fe177fff640 10 monclient(hunting): _negotiate 2024-03-29T10:36:59.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.184+0000 7fe177fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:36:59.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.184+0000 7fe177fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:36:59.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.184+0000 7fe177fff640 10 monclient(hunting): my global_id is 4308 2024-03-29T10:36:59.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.184+0000 7fe177fff640 1 -- 172.21.15.84:0/2986533734 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1900ae300 con 0x7fe19006ed60 2024-03-29T10:36:59.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.185+0000 7fe177fff640 1 -- 172.21.15.84:0/2986533734 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2208457323 0 0) 0x7fe17c004eb0 con 0x7fe19006ade0 2024-03-29T10:36:59.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.185+0000 7fe177fff640 1 -- 172.21.15.84:0/2986533734 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1900ae0b0 con 0x7fe19006ade0 2024-03-29T10:36:59.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.185+0000 7fe177fff640 1 -- 172.21.15.84:0/2986533734 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fe180001630 con 0x7fe19006ade0 2024-03-29T10:36:59.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.185+0000 7fe177fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:59.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.185+0000 7fe177fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:36:59.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.185+0000 7fe177fff640 10 monclient(hunting): dump: 2024-03-29T10:36:59.173 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:59.173 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:59.173 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:59.173 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:59.173 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:59.173 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:59.173 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:59.173 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:59.174 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:59.174 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:59.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.185+0000 7fe177fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:36:59.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.186+0000 7fe177fff640 1 -- 172.21.15.84:0/2986533734 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 796480826 0 0) 0x7fe1900ae300 con 0x7fe19006ed60 2024-03-29T10:36:59.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.186+0000 7fe177fff640 1 -- 172.21.15.84:0/2986533734 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe17c004eb0 con 0x7fe19006ed60 2024-03-29T10:36:59.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.186+0000 7fe177fff640 1 -- 172.21.15.84:0/2986533734 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 212214682 0 0) 0x7fe1900ae0b0 con 0x7fe19006ade0 2024-03-29T10:36:59.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.186+0000 7fe177fff640 1 -- 172.21.15.84:0/2986533734 >> v1:172.21.15.86:6789/0 conn(0x7fe190072340 legacy=0x7fe1901ee550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:59.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.186+0000 7fe177fff640 1 -- 172.21.15.84:0/2986533734 >> v1:172.21.15.84:6789/0 conn(0x7fe19006ed60 legacy=0x7fe1900ace30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:59.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.186+0000 7fe177fff640 10 monclient: _finish_hunting 0 2024-03-29T10:36:59.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.186+0000 7fe177fff640 1 monclient: found mon.c 2024-03-29T10:36:59.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.186+0000 7fe177fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:36:59.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.186+0000 7fe177fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:36:59.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.186+0000 7fe177fff640 1 -- 172.21.15.84:0/2986533734 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1900ad540 con 0x7fe19006ade0 2024-03-29T10:36:59.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.186+0000 7fe177fff640 10 monclient: _finish_auth 0 2024-03-29T10:36:59.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.186+0000 7fe177fff640 10 monclient: _check_auth_tickets 2024-03-29T10:36:59.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.186+0000 7fe177fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:36:59.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.187+0000 7fe19581e640 5 monclient: authenticate success, global_id 4322 2024-03-29T10:36:59.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.187+0000 7fe19581e640 10 monclient: _renew_subs 2024-03-29T10:36:59.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.187+0000 7fe19581e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:36:59.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.187+0000 7fe19581e640 1 -- 172.21.15.84:0/2986533734 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe190142010 con 0x7fe19006ade0 2024-03-29T10:36:59.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.187+0000 7fe19581e640 10 client.4322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:36:59.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.187+0000 7fe19581e640 10 monclient: _renew_subs 2024-03-29T10:36:59.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.187+0000 7fe19581e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:36:59.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.187+0000 7fe19581e640 1 -- 172.21.15.84:0/2986533734 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1900ac690 con 0x7fe19006ade0 2024-03-29T10:36:59.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.187+0000 7fe19581e640 1 librados: init done 2024-03-29T10:36:59.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.190+0000 7fe175ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:59.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.190+0000 7fe175ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:59.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.190+0000 7fe175ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:36:59.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.190+0000 7fe175ffb640 1 -- 172.21.15.84:0/2986533734 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe190142360 con 0x7fe19006ade0 2024-03-29T10:36:59.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.191+0000 7fe177fff640 1 -- 172.21.15.84:0/2986533734 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe180004c70 con 0x7fe19006ade0 2024-03-29T10:36:59.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.191+0000 7fe177fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:36:59.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.191+0000 7fe177fff640 1 -- 172.21.15.84:0/2986533734 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fe180004f80 con 0x7fe19006ade0 2024-03-29T10:36:59.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.191+0000 7fe177fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:36:59.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.191+0000 7fe177fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:36:59.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.191+0000 7fe177fff640 10 monclient: dump: 2024-03-29T10:36:59.177 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:36:59.177 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:36:59.177 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:59.177 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:36:59.177 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:36:59.177 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:36:59.177 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:36:59.177 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:36:59.177 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:36:59.177 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:36:59.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.193+0000 7fe177fff640 1 -- 172.21.15.84:0/2986533734 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7fe18001af50 con 0x7fe19006ade0 2024-03-29T10:36:59.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.193+0000 7fe18d7fa640 4 set_mon_vals no callback set 2024-03-29T10:36:59.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.194+0000 7fe177fff640 1 -- 172.21.15.84:0/2986533734 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7fe180084ce0 con 0x7fe19006ade0 2024-03-29T10:36:59.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.195+0000 7fe177fff640 10 client.4322.objecter ms_dispatch 0x7fe190000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:36:59.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.195+0000 7fe177fff640 3 client.4322.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:36:59.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.195+0000 7fe177fff640 3 client.4322.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:36:59.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.195+0000 7fe177fff640 20 client.4322.objecter dump_active .. 0 homeless 2024-03-29T10:36:59.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.196+0000 7fe18e7fc640 10 monclient: get_auth_request con 0x7fe17c06de20 auth_method 0 2024-03-29T10:36:59.181 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.196+0000 7fe177fff640 10 client.4322.objecter ms_handle_connect 0x7fe17c06de20 2024-03-29T10:36:59.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.202+0000 7fe177fff640 1 -- 172.21.15.84:0/2986533734 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe1800505c0 con 0x7fe19006ade0 2024-03-29T10:36:59.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.202+0000 7fe177fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:36:59.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.202+0000 7fe177fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:36:59.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.523+0000 7fe175ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:59.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.523+0000 7fe175ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:59.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.523+0000 7fe175ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:36:59.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.523+0000 7fe175ffb640 1 -- 172.21.15.84:0/2986533734 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe19006b1c0 con 0x7fe19006ade0 2024-03-29T10:36:59.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.525+0000 7fe177fff640 1 -- 172.21.15.84:0/2986533734 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7fe1800524f0 con 0x7fe19006ade0 2024-03-29T10:36:59.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.525+0000 7fe177fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:36:59.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.525+0000 7fe177fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:36:59.512 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:36:59.512 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:36:59.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.531+0000 7fe175ffb640 10 librados: watch_flush enter 2024-03-29T10:36:59.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.535+0000 7fe175ffb640 10 librados: watch_flush exit 2024-03-29T10:36:59.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.535+0000 7fe175ffb640 20 client.4322.objecter shutdown clearing up homeless session... 2024-03-29T10:36:59.520 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.535+0000 7fe175ffb640 10 client.4322.objecter successfully canceled tick 2024-03-29T10:36:59.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.536+0000 7fe175ffb640 1 -- 172.21.15.84:0/2986533734 >> v1:172.21.15.86:6816/3337225337 conn(0x7fe17c06de20 legacy=0x7fe17c0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:59.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.536+0000 7fe175ffb640 10 monclient: shutdown 2024-03-29T10:36:59.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.536+0000 7fe175ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:36:59.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.536+0000 7fe175ffb640 1 -- 172.21.15.84:0/2986533734 >> v1:172.21.15.84:6790/0 conn(0x7fe19006ade0 legacy=0x7fe1900a9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:36:59.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.537+0000 7fe175ffb640 1 -- 172.21.15.84:0/2986533734 shutdown_connections 2024-03-29T10:36:59.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.537+0000 7fe175ffb640 1 -- 172.21.15.84:0/2986533734 >> 172.21.15.84:0/2986533734 conn(0x7fe1900b4930 msgr2=0x7fe1900afa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:36:59.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.537+0000 7fe175ffb640 1 -- 172.21.15.84:0/2986533734 shutdown_connections 2024-03-29T10:36:59.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.537+0000 7fe175ffb640 1 -- 172.21.15.84:0/2986533734 wait complete. 2024-03-29T10:36:59.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:36:59.537+0000 7fe175ffb640 1 librados: shutdown 2024-03-29T10:36:59.539 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:37:06.345 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:37:06.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.504+0000 7ffb394c3640 10 monclient: get_monmap_and_config 2024-03-29T10:37:06.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.504+0000 7ffb394c3640 10 monclient: build_initial_monmap 2024-03-29T10:37:06.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.504+0000 7ffb394c3640 10 monclient: monmap: 2024-03-29T10:37:06.490 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:37:06.490 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:06.490 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:37:06.505919+0000 2024-03-29T10:37:06.490 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:37:06.505919+0000 2024-03-29T10:37:06.490 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:06.490 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:06.490 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:37:06.490 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:37:06.490 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:37:06.490 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:06.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.506+0000 7ffb394c3640 1 Processor -- start 2024-03-29T10:37:06.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.506+0000 7ffb394c3640 1 -- start start 2024-03-29T10:37:06.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.506+0000 7ffb394c3640 10 monclient: init 2024-03-29T10:37:06.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.507+0000 7ffb394c3640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:06.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.507+0000 7ffb394c3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:37:06.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.507+0000 7ffb394c3640 10 monclient(hunting): picked mon.noname-a con 0x7ffb341457f0 addr v1:172.21.15.84:6789/0 2024-03-29T10:37:06.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.507+0000 7ffb394c3640 10 monclient(hunting): picked mon.noname-b con 0x7ffb3413f650 addr v1:172.21.15.84:6790/0 2024-03-29T10:37:06.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.507+0000 7ffb394c3640 10 monclient(hunting): picked mon.noname-c con 0x7ffb340a9d70 addr v1:172.21.15.86:6789/0 2024-03-29T10:37:06.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.507+0000 7ffb394c3640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb340aaef0 con 0x7ffb341457f0 2024-03-29T10:37:06.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.507+0000 7ffb394c3640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb34145bd0 con 0x7ffb340a9d70 2024-03-29T10:37:06.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.507+0000 7ffb394c3640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb34145eb0 con 0x7ffb3413f650 2024-03-29T10:37:06.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.507+0000 7ffb394c3640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:06.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.507+0000 7ffb394c3640 10 monclient(hunting): authenticate will time out at 928.770476s 2024-03-29T10:37:06.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.507+0000 7ffb32ffd640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7ffb341457f0 0x7ffb3413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:38728/0 (socket says 172.21.15.84:38728) 2024-03-29T10:37:06.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.507+0000 7ffb32ffd640 1 -- 172.21.15.84:0/3429962860 learned_addr learned my addr 172.21.15.84:0/3429962860 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:37:06.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.508+0000 7ffb327fc640 1 -- 172.21.15.84:0/3429962860 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1582001968 0 0) 0x7ffb340aaef0 con 0x7ffb341457f0 2024-03-29T10:37:06.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.508+0000 7ffb327fc640 10 monclient(hunting): _negotiate 2024-03-29T10:37:06.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.508+0000 7ffb327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:06.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.508+0000 7ffb327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:06.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.508+0000 7ffb327fc640 10 monclient(hunting): my global_id is 4311 2024-03-29T10:37:06.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.508+0000 7ffb327fc640 1 -- 172.21.15.84:0/3429962860 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb20003520 con 0x7ffb341457f0 2024-03-29T10:37:06.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.508+0000 7ffb327fc640 1 -- 172.21.15.84:0/3429962860 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1632283467 0 0) 0x7ffb20003520 con 0x7ffb341457f0 2024-03-29T10:37:06.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.508+0000 7ffb327fc640 1 -- 172.21.15.84:0/3429962860 >> v1:172.21.15.84:6790/0 conn(0x7ffb3413f650 legacy=0x7ffb340b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:06.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.508+0000 7ffb327fc640 1 -- 172.21.15.84:0/3429962860 >> v1:172.21.15.86:6789/0 conn(0x7ffb340a9d70 legacy=0x7ffb3414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:37:06.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.508+0000 7ffb327fc640 10 monclient: _finish_hunting 0 2024-03-29T10:37:06.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.508+0000 7ffb327fc640 1 monclient: found mon.noname-a 2024-03-29T10:37:06.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.508+0000 7ffb327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:06.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.508+0000 7ffb327fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T10:37:06.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.508+0000 7ffb327fc640 1 -- 172.21.15.84:0/3429962860 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb34146340 con 0x7ffb341457f0 2024-03-29T10:37:06.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.508+0000 7ffb327fc640 10 monclient: _finish_auth 0 2024-03-29T10:37:06.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.508+0000 7ffb327fc640 10 monclient: _check_auth_tickets 2024-03-29T10:37:06.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.508+0000 7ffb327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:06.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.508+0000 7ffb394c3640 5 monclient: authenticate success, global_id 4311 2024-03-29T10:37:06.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.508+0000 7ffb394c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:37:06.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.508+0000 7ffb327fc640 1 -- 172.21.15.84:0/3429962860 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7ffb24001690 con 0x7ffb341457f0 2024-03-29T10:37:06.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.508+0000 7ffb327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:06.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.508+0000 7ffb327fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:37:06.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.508+0000 7ffb327fc640 10 monclient: dump: 2024-03-29T10:37:06.496 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:06.496 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:06.496 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:06.496 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:06.496 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:06.496 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:06.496 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:06.496 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:06.496 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:06.496 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:06.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.508+0000 7ffb394c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:37:06.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.508+0000 7ffb327fc640 1 -- 172.21.15.84:0/3429962860 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb24001860 con 0x7ffb341457f0 2024-03-29T10:37:06.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.508+0000 7ffb327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:06.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.509+0000 7ffb394c3640 10 monclient: get_monmap_and_config success 2024-03-29T10:37:06.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.509+0000 7ffb394c3640 4 set_mon_vals no callback set 2024-03-29T10:37:06.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.509+0000 7ffb327fc640 1 -- 172.21.15.84:0/3429962860 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7ffb24001c70 con 0x7ffb341457f0 2024-03-29T10:37:06.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.509+0000 7ffb327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:06.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.509+0000 7ffb327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:37:06.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.509+0000 7ffb327fc640 10 monclient: dump: 2024-03-29T10:37:06.497 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:06.497 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:06.497 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:06.497 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:06.497 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:06.498 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:06.498 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:06.498 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:06.498 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:06.498 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:06.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.509+0000 7ffb394c3640 10 monclient: shutdown 2024-03-29T10:37:06.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.509+0000 7ffb394c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:06.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.509+0000 7ffb394c3640 1 -- 172.21.15.84:0/3429962860 >> v1:172.21.15.84:6789/0 conn(0x7ffb341457f0 legacy=0x7ffb3413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:06.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.509+0000 7ffb394c3640 1 -- 172.21.15.84:0/3429962860 shutdown_connections 2024-03-29T10:37:06.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.509+0000 7ffb394c3640 1 -- 172.21.15.84:0/3429962860 >> 172.21.15.84:0/3429962860 conn(0x7ffb34066a00 msgr2=0x7ffb34066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:06.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.510+0000 7ffb394c3640 1 -- 172.21.15.84:0/3429962860 shutdown_connections 2024-03-29T10:37:06.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.510+0000 7ffb394c3640 1 -- 172.21.15.84:0/3429962860 wait complete. 2024-03-29T10:37:06.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.511+0000 7ffb394c3640 10 monclient: build_initial_monmap 2024-03-29T10:37:06.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.511+0000 7ffb394c3640 10 monclient: monmap: 2024-03-29T10:37:06.503 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:37:06.503 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:37:06.503 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:37:06.503 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:37:06.503 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:06.503 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:06.503 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:37:06.503 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:37:06.503 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:37:06.503 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:06.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.511+0000 7ffb394c3640 1 librados: starting msgr at 2024-03-29T10:37:06.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.511+0000 7ffb394c3640 1 librados: starting objecter 2024-03-29T10:37:06.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.511+0000 7ffb394c3640 1 Processor -- start 2024-03-29T10:37:06.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.511+0000 7ffb394c3640 1 -- start start 2024-03-29T10:37:06.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.511+0000 7ffb394c3640 1 librados: setting wanted keys 2024-03-29T10:37:06.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.511+0000 7ffb394c3640 1 librados: calling monclient init 2024-03-29T10:37:06.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.511+0000 7ffb394c3640 10 monclient: init 2024-03-29T10:37:06.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.512+0000 7ffb394c3640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:06.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.512+0000 7ffb394c3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:37:06.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.512+0000 7ffb394c3640 10 monclient(hunting): picked mon.noname-c con 0x7ffb340a9d70 addr v1:172.21.15.84:6790/0 2024-03-29T10:37:06.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.512+0000 7ffb394c3640 10 monclient(hunting): picked mon.noname-b con 0x7ffb3413f650 addr v1:172.21.15.86:6789/0 2024-03-29T10:37:06.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.512+0000 7ffb394c3640 10 monclient(hunting): picked mon.noname-a con 0x7ffb340abd10 addr v1:172.21.15.84:6789/0 2024-03-29T10:37:06.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.512+0000 7ffb394c3640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb340ad750 con 0x7ffb340abd10 2024-03-29T10:37:06.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.512+0000 7ffb394c3640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb341db250 con 0x7ffb3413f650 2024-03-29T10:37:06.504 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.512+0000 7ffb394c3640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb341db770 con 0x7ffb340a9d70 2024-03-29T10:37:06.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.512+0000 7ffb394c3640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:06.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.512+0000 7ffb394c3640 10 monclient(hunting): authenticate will time out at 928.775844s 2024-03-29T10:37:06.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.512+0000 7ffb32ffd640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7ffb340a9d70 0x7ffb340ab600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:50246/0 (socket says 172.21.15.84:50246) 2024-03-29T10:37:06.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.513+0000 7ffb32ffd640 1 -- 172.21.15.84:0/1841708805 learned_addr learned my addr 172.21.15.84:0/1841708805 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:37:06.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.513+0000 7ffb2b7fe640 10 client.?.objecter ms_handle_connect 0x7ffb340a9d70 2024-03-29T10:37:06.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.513+0000 7ffb2b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:06.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.513+0000 7ffb2b7fe640 1 -- 172.21.15.84:0/1841708805 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070238470 0 0) 0x7ffb341db770 con 0x7ffb340a9d70 2024-03-29T10:37:06.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.513+0000 7ffb2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:37:06.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.513+0000 7ffb2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:06.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.513+0000 7ffb2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:06.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.513+0000 7ffb2b7fe640 10 monclient(hunting): my global_id is 4328 2024-03-29T10:37:06.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.513+0000 7ffb2b7fe640 1 -- 172.21.15.84:0/1841708805 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb14004eb0 con 0x7ffb340a9d70 2024-03-29T10:37:06.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.513+0000 7ffb2b7fe640 10 client.?.objecter ms_handle_connect 0x7ffb3413f650 2024-03-29T10:37:06.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.513+0000 7ffb2b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:06.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.513+0000 7ffb2b7fe640 1 -- 172.21.15.84:0/1841708805 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2944675520 0 0) 0x7ffb341db250 con 0x7ffb3413f650 2024-03-29T10:37:06.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.513+0000 7ffb2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:37:06.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.513+0000 7ffb2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:06.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.513+0000 7ffb2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:06.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.513+0000 7ffb2b7fe640 10 monclient(hunting): my global_id is 4300 2024-03-29T10:37:06.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.513+0000 7ffb2b7fe640 1 -- 172.21.15.84:0/1841708805 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb341db770 con 0x7ffb3413f650 2024-03-29T10:37:06.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.514+0000 7ffb2b7fe640 10 client.?.objecter ms_handle_connect 0x7ffb340abd10 2024-03-29T10:37:06.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.514+0000 7ffb2b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:06.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.514+0000 7ffb2b7fe640 1 -- 172.21.15.84:0/1841708805 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4286687507 0 0) 0x7ffb14004eb0 con 0x7ffb340a9d70 2024-03-29T10:37:06.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.514+0000 7ffb2b7fe640 1 -- 172.21.15.84:0/1841708805 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb341db250 con 0x7ffb340a9d70 2024-03-29T10:37:06.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.514+0000 7ffb2b7fe640 1 -- 172.21.15.84:0/1841708805 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7ffb24001b70 con 0x7ffb340a9d70 2024-03-29T10:37:06.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.514+0000 7ffb2b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:06.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.514+0000 7ffb2b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:37:06.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.514+0000 7ffb2b7fe640 10 monclient(hunting): dump: 2024-03-29T10:37:06.507 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:06.507 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:06.507 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:06.507 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:06.507 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:06.507 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:06.507 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:06.507 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:06.507 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:06.507 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:06.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.514+0000 7ffb2b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:37:06.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.514+0000 7ffb2b7fe640 1 -- 172.21.15.84:0/1841708805 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4213507207 0 0) 0x7ffb341db250 con 0x7ffb340a9d70 2024-03-29T10:37:06.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.514+0000 7ffb2b7fe640 1 -- 172.21.15.84:0/1841708805 >> v1:172.21.15.86:6789/0 conn(0x7ffb3413f650 legacy=0x7ffb34137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:06.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.514+0000 7ffb2b7fe640 1 -- 172.21.15.84:0/1841708805 >> v1:172.21.15.84:6789/0 conn(0x7ffb340abd10 legacy=0x7ffb341e2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:06.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.514+0000 7ffb2b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:37:06.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.514+0000 7ffb2b7fe640 1 monclient: found mon.c 2024-03-29T10:37:06.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.514+0000 7ffb2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:06.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.514+0000 7ffb2b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:37:06.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.514+0000 7ffb2b7fe640 1 -- 172.21.15.84:0/1841708805 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb340b0c40 con 0x7ffb340a9d70 2024-03-29T10:37:06.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.515+0000 7ffb2b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:37:06.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.515+0000 7ffb2b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:37:06.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.515+0000 7ffb2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:06.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.515+0000 7ffb394c3640 5 monclient: authenticate success, global_id 4328 2024-03-29T10:37:06.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.515+0000 7ffb394c3640 10 monclient: _renew_subs 2024-03-29T10:37:06.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.515+0000 7ffb394c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:37:06.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.515+0000 7ffb394c3640 1 -- 172.21.15.84:0/1841708805 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb340b0e90 con 0x7ffb340a9d70 2024-03-29T10:37:06.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.515+0000 7ffb394c3640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:37:06.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.515+0000 7ffb394c3640 10 monclient: _renew_subs 2024-03-29T10:37:06.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.515+0000 7ffb394c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:37:06.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.515+0000 7ffb394c3640 1 -- 172.21.15.84:0/1841708805 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb341dae30 con 0x7ffb340a9d70 2024-03-29T10:37:06.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.515+0000 7ffb394c3640 1 librados: init done 2024-03-29T10:37:06.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.515+0000 7ffb2b7fe640 1 -- 172.21.15.84:0/1841708805 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb24004bf0 con 0x7ffb340a9d70 2024-03-29T10:37:06.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.515+0000 7ffb2b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:06.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.515+0000 7ffb2b7fe640 1 -- 172.21.15.84:0/1841708805 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7ffb24004f00 con 0x7ffb340a9d70 2024-03-29T10:37:06.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.515+0000 7ffb2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:06.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.515+0000 7ffb2b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:37:06.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.515+0000 7ffb2b7fe640 10 monclient: dump: 2024-03-29T10:37:06.509 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:06.509 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:06.509 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:06.509 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:06.509 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:06.509 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:06.509 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:06.510 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:06.510 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:06.510 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:06.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.515+0000 7ffb31ffb640 4 set_mon_vals no callback set 2024-03-29T10:37:06.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.516+0000 7ffb394c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:06.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.516+0000 7ffb394c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:06.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.516+0000 7ffb394c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:37:06.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.516+0000 7ffb394c3640 1 -- 172.21.15.84:0/1841708805 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb18005080 con 0x7ffb340a9d70 2024-03-29T10:37:06.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.518+0000 7ffb2b7fe640 1 -- 172.21.15.84:0/1841708805 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7ffb2401aed0 con 0x7ffb340a9d70 2024-03-29T10:37:06.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.519+0000 7ffb2b7fe640 1 -- 172.21.15.84:0/1841708805 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7ffb24084b40 con 0x7ffb340a9d70 2024-03-29T10:37:06.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.519+0000 7ffb2b7fe640 10 client.4328.objecter ms_dispatch 0x7ffb34000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:37:06.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.519+0000 7ffb2b7fe640 3 client.4328.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:37:06.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.519+0000 7ffb2b7fe640 3 client.4328.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:37:06.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.519+0000 7ffb2b7fe640 20 client.4328.objecter dump_active .. 0 homeless 2024-03-29T10:37:06.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.520+0000 7ffb2bfff640 10 monclient: get_auth_request con 0x7ffb1406da60 auth_method 0 2024-03-29T10:37:06.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.521+0000 7ffb2b7fe640 10 client.4328.objecter ms_handle_connect 0x7ffb1406da60 2024-03-29T10:37:06.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.526+0000 7ffb2b7fe640 1 -- 172.21.15.84:0/1841708805 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffb24050420 con 0x7ffb340a9d70 2024-03-29T10:37:06.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.526+0000 7ffb2b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:06.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.526+0000 7ffb2b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:37:06.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.846+0000 7ffb394c3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:06.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.846+0000 7ffb394c3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:06.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.846+0000 7ffb394c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:37:06.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.846+0000 7ffb394c3640 1 -- 172.21.15.84:0/1841708805 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb18005330 con 0x7ffb340a9d70 2024-03-29T10:37:06.833 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:37:06.833 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:37:06.833 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.848+0000 7ffb2b7fe640 1 -- 172.21.15.84:0/1841708805 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7ffb24052350 con 0x7ffb340a9d70 2024-03-29T10:37:06.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.848+0000 7ffb2b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:06.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.848+0000 7ffb2b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:37:06.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.850+0000 7ffb394c3640 10 librados: watch_flush enter 2024-03-29T10:37:06.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.851+0000 7ffb394c3640 10 librados: watch_flush exit 2024-03-29T10:37:06.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.851+0000 7ffb394c3640 20 client.4328.objecter shutdown clearing up homeless session... 2024-03-29T10:37:06.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.851+0000 7ffb394c3640 10 client.4328.objecter successfully canceled tick 2024-03-29T10:37:06.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.851+0000 7ffb394c3640 1 -- 172.21.15.84:0/1841708805 >> v1:172.21.15.86:6816/3337225337 conn(0x7ffb1406da60 legacy=0x7ffb1406ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:06.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.851+0000 7ffb394c3640 10 monclient: shutdown 2024-03-29T10:37:06.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.851+0000 7ffb394c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:06.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.851+0000 7ffb394c3640 1 -- 172.21.15.84:0/1841708805 >> v1:172.21.15.84:6790/0 conn(0x7ffb340a9d70 legacy=0x7ffb340ab600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:06.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.852+0000 7ffb394c3640 1 -- 172.21.15.84:0/1841708805 shutdown_connections 2024-03-29T10:37:06.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.852+0000 7ffb394c3640 1 -- 172.21.15.84:0/1841708805 >> 172.21.15.84:0/1841708805 conn(0x7ffb34066a00 msgr2=0x7ffb34066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:06.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.853+0000 7ffb394c3640 1 -- 172.21.15.84:0/1841708805 shutdown_connections 2024-03-29T10:37:06.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.853+0000 7ffb394c3640 1 -- 172.21.15.84:0/1841708805 wait complete. 2024-03-29T10:37:06.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:06.855+0000 7ffb394c3640 1 librados: shutdown 2024-03-29T10:37:06.854 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:37:13.661 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:37:13.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.825+0000 7fe9f9699640 10 monclient: get_monmap_and_config 2024-03-29T10:37:13.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.825+0000 7fe9f9699640 10 monclient: build_initial_monmap 2024-03-29T10:37:13.810 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.825+0000 7fe9f9699640 10 monclient: monmap: 2024-03-29T10:37:13.810 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:37:13.810 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:13.810 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:37:13.826300+0000 2024-03-29T10:37:13.810 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:37:13.826300+0000 2024-03-29T10:37:13.810 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:13.810 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:13.810 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:37:13.811 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:37:13.811 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:37:13.811 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:13.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.826+0000 7fe9f9699640 1 Processor -- start 2024-03-29T10:37:13.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.826+0000 7fe9f9699640 1 -- start start 2024-03-29T10:37:13.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.826+0000 7fe9f9699640 10 monclient: init 2024-03-29T10:37:13.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.827+0000 7fe9f9699640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:13.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.827+0000 7fe9f9699640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:37:13.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.827+0000 7fe9f9699640 10 monclient(hunting): picked mon.noname-a con 0x7fe9f40b4920 addr v1:172.21.15.84:6789/0 2024-03-29T10:37:13.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.827+0000 7fe9f9699640 10 monclient(hunting): picked mon.noname-b con 0x7fe9f40b4200 addr v1:172.21.15.84:6790/0 2024-03-29T10:37:13.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.827+0000 7fe9f9699640 10 monclient(hunting): picked mon.noname-c con 0x7fe9f413f9a0 addr v1:172.21.15.86:6789/0 2024-03-29T10:37:13.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.827+0000 7fe9f9699640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f40b0930 con 0x7fe9f40b4920 2024-03-29T10:37:13.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.827+0000 7fe9f9699640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f40af460 con 0x7fe9f413f9a0 2024-03-29T10:37:13.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.827+0000 7fe9f9699640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f40aeda0 con 0x7fe9f40b4200 2024-03-29T10:37:13.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.827+0000 7fe9f9699640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:13.812 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.827+0000 7fe9f9699640 10 monclient(hunting): authenticate will time out at 936.090579s 2024-03-29T10:37:13.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.827+0000 7fe9f8e98640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fe9f413f9a0 0x7fe9f414d8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:43280/0 (socket says 172.21.15.84:43280) 2024-03-29T10:37:13.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.827+0000 7fe9f8e98640 1 -- 172.21.15.84:0/1198742960 learned_addr learned my addr 172.21.15.84:0/1198742960 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:37:13.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f2ffd640 1 -- 172.21.15.84:0/1198742960 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2253026516 0 0) 0x7fe9f40aeda0 con 0x7fe9f40b4200 2024-03-29T10:37:13.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:37:13.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:13.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:13.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f2ffd640 10 monclient(hunting): my global_id is 4331 2024-03-29T10:37:13.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f2ffd640 1 -- 172.21.15.84:0/1198742960 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9dc003520 con 0x7fe9f40b4200 2024-03-29T10:37:13.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f2ffd640 1 -- 172.21.15.84:0/1198742960 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 54972790 0 0) 0x7fe9f40b0930 con 0x7fe9f40b4920 2024-03-29T10:37:13.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:37:13.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:13.814 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:13.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f2ffd640 10 monclient(hunting): my global_id is 4317 2024-03-29T10:37:13.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f2ffd640 1 -- 172.21.15.84:0/1198742960 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9f40aeda0 con 0x7fe9f40b4920 2024-03-29T10:37:13.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f2ffd640 1 -- 172.21.15.84:0/1198742960 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2750235357 0 0) 0x7fe9dc003520 con 0x7fe9f40b4200 2024-03-29T10:37:13.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f2ffd640 1 -- 172.21.15.84:0/1198742960 >> v1:172.21.15.86:6789/0 conn(0x7fe9f413f9a0 legacy=0x7fe9f414d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:13.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f2ffd640 1 -- 172.21.15.84:0/1198742960 >> v1:172.21.15.84:6789/0 conn(0x7fe9f40b4920 legacy=0x7fe9f413f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:13.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:37:13.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f2ffd640 1 monclient: found mon.noname-b 2024-03-29T10:37:13.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:13.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:37:13.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f2ffd640 1 -- 172.21.15.84:0/1198742960 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f40af980 con 0x7fe9f40b4200 2024-03-29T10:37:13.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:37:13.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:37:13.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:13.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f9699640 5 monclient: authenticate success, global_id 4331 2024-03-29T10:37:13.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f9699640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:37:13.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f2ffd640 1 -- 172.21.15.84:0/1198742960 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fe9e8001690 con 0x7fe9f40b4200 2024-03-29T10:37:13.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:13.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:37:13.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f2ffd640 10 monclient: dump: 2024-03-29T10:37:13.816 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:13.816 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:13.816 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:13.816 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:13.816 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:13.816 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:13.816 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:13.816 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:13.816 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:13.816 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:13.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.828+0000 7fe9f9699640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:37:13.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.829+0000 7fe9f2ffd640 1 -- 172.21.15.84:0/1198742960 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9e80017f0 con 0x7fe9f40b4200 2024-03-29T10:37:13.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.829+0000 7fe9f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:13.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.829+0000 7fe9f9699640 10 monclient: get_monmap_and_config success 2024-03-29T10:37:13.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.829+0000 7fe9f9699640 4 set_mon_vals no callback set 2024-03-29T10:37:13.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.829+0000 7fe9f2ffd640 1 -- 172.21.15.84:0/1198742960 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fe9e8001c00 con 0x7fe9f40b4200 2024-03-29T10:37:13.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.829+0000 7fe9f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:13.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.829+0000 7fe9f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:37:13.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.829+0000 7fe9f2ffd640 10 monclient: dump: 2024-03-29T10:37:13.817 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:13.817 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:13.817 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:13.817 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:13.817 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:13.817 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:13.818 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:13.818 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:13.818 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:13.818 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:13.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.829+0000 7fe9f9699640 10 monclient: shutdown 2024-03-29T10:37:13.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.830+0000 7fe9f9699640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:13.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.830+0000 7fe9f9699640 1 -- 172.21.15.84:0/1198742960 >> v1:172.21.15.84:6790/0 conn(0x7fe9f40b4200 legacy=0x7fe9f40b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:13.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.830+0000 7fe9f9699640 1 -- 172.21.15.84:0/1198742960 shutdown_connections 2024-03-29T10:37:13.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.830+0000 7fe9f9699640 1 -- 172.21.15.84:0/1198742960 >> 172.21.15.84:0/1198742960 conn(0x7fe9f406d230 msgr2=0x7fe9f4066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:13.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.830+0000 7fe9f9699640 1 -- 172.21.15.84:0/1198742960 shutdown_connections 2024-03-29T10:37:13.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.830+0000 7fe9f9699640 1 -- 172.21.15.84:0/1198742960 wait complete. 2024-03-29T10:37:13.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.831+0000 7fe9f9699640 10 monclient: build_initial_monmap 2024-03-29T10:37:13.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.831+0000 7fe9f9699640 10 monclient: monmap: 2024-03-29T10:37:13.822 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:37:13.822 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:37:13.822 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:37:13.822 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:37:13.823 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:13.823 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:13.823 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:37:13.823 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:37:13.823 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:37:13.823 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:13.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.831+0000 7fe9f9699640 1 librados: starting msgr at 2024-03-29T10:37:13.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.831+0000 7fe9f9699640 1 librados: starting objecter 2024-03-29T10:37:13.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.831+0000 7fe9f9699640 1 Processor -- start 2024-03-29T10:37:13.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.832+0000 7fe9f9699640 1 -- start start 2024-03-29T10:37:13.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.832+0000 7fe9f9699640 1 librados: setting wanted keys 2024-03-29T10:37:13.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.832+0000 7fe9f9699640 1 librados: calling monclient init 2024-03-29T10:37:13.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.832+0000 7fe9f9699640 10 monclient: init 2024-03-29T10:37:13.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.833+0000 7fe9f9699640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:13.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.833+0000 7fe9f9699640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:37:13.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.833+0000 7fe9f9699640 10 monclient(hunting): picked mon.noname-a con 0x7fe9f40b4200 addr v1:172.21.15.84:6789/0 2024-03-29T10:37:13.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.833+0000 7fe9f9699640 10 monclient(hunting): picked mon.noname-b con 0x7fe9f40b4920 addr v1:172.21.15.86:6789/0 2024-03-29T10:37:13.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.833+0000 7fe9f9699640 10 monclient(hunting): picked mon.noname-c con 0x7fe9f413f9a0 addr v1:172.21.15.84:6790/0 2024-03-29T10:37:13.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.833+0000 7fe9f9699640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f41d2c20 con 0x7fe9f40b4200 2024-03-29T10:37:13.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.833+0000 7fe9f9699640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f41d3850 con 0x7fe9f40b4920 2024-03-29T10:37:13.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.833+0000 7fe9f9699640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f41d39e0 con 0x7fe9f413f9a0 2024-03-29T10:37:13.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.833+0000 7fe9f9699640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:13.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.833+0000 7fe9f9699640 10 monclient(hunting): authenticate will time out at 936.096385s 2024-03-29T10:37:13.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.834+0000 7fe9f3fff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fe9f40b4200 0x7fe9f4071b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:49008/0 (socket says 172.21.15.84:49008) 2024-03-29T10:37:13.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.834+0000 7fe9f3fff640 1 -- 172.21.15.84:0/2021540858 learned_addr learned my addr 172.21.15.84:0/2021540858 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:37:13.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.834+0000 7fe9f0ff9640 10 client.?.objecter ms_handle_connect 0x7fe9f413f9a0 2024-03-29T10:37:13.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.834+0000 7fe9f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:13.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.834+0000 7fe9f0ff9640 10 client.?.objecter ms_handle_connect 0x7fe9f40b4200 2024-03-29T10:37:13.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.834+0000 7fe9f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:13.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.834+0000 7fe9f0ff9640 1 -- 172.21.15.84:0/2021540858 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 795425094 0 0) 0x7fe9f41d2c20 con 0x7fe9f40b4200 2024-03-29T10:37:13.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.834+0000 7fe9f0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:37:13.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.834+0000 7fe9f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:13.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.834+0000 7fe9f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:13.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.834+0000 7fe9f0ff9640 10 monclient(hunting): my global_id is 4320 2024-03-29T10:37:13.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.834+0000 7fe9f0ff9640 1 -- 172.21.15.84:0/2021540858 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9e0004eb0 con 0x7fe9f40b4200 2024-03-29T10:37:13.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.835+0000 7fe9f0ff9640 1 -- 172.21.15.84:0/2021540858 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1018836481 0 0) 0x7fe9f41d39e0 con 0x7fe9f413f9a0 2024-03-29T10:37:13.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.835+0000 7fe9f0ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:37:13.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.835+0000 7fe9f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:13.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.835+0000 7fe9f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:13.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.835+0000 7fe9f0ff9640 10 monclient(hunting): my global_id is 4334 2024-03-29T10:37:13.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.835+0000 7fe9f0ff9640 1 -- 172.21.15.84:0/2021540858 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9f41d2c20 con 0x7fe9f413f9a0 2024-03-29T10:37:13.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.835+0000 7fe9f0ff9640 1 -- 172.21.15.84:0/2021540858 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3002173304 0 0) 0x7fe9e0004eb0 con 0x7fe9f40b4200 2024-03-29T10:37:13.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.835+0000 7fe9f0ff9640 1 -- 172.21.15.84:0/2021540858 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9f41d39e0 con 0x7fe9f40b4200 2024-03-29T10:37:13.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.835+0000 7fe9f0ff9640 1 -- 172.21.15.84:0/2021540858 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fe9e4001a90 con 0x7fe9f40b4200 2024-03-29T10:37:13.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.836+0000 7fe9f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:13.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.836+0000 7fe9f0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:37:13.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.836+0000 7fe9f0ff9640 10 monclient(hunting): dump: 2024-03-29T10:37:13.826 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:13.826 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:13.826 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:13.826 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:13.826 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:13.826 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:13.826 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:13.826 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:13.826 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:13.826 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:13.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.836+0000 7fe9f0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:37:13.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.836+0000 7fe9f0ff9640 10 client.?.objecter ms_handle_connect 0x7fe9f40b4920 2024-03-29T10:37:13.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.837+0000 7fe9f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:13.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.837+0000 7fe9f0ff9640 1 -- 172.21.15.84:0/2021540858 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 51468024 0 0) 0x7fe9f41d2c20 con 0x7fe9f413f9a0 2024-03-29T10:37:13.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.837+0000 7fe9f0ff9640 1 -- 172.21.15.84:0/2021540858 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9e0004eb0 con 0x7fe9f413f9a0 2024-03-29T10:37:13.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.837+0000 7fe9f0ff9640 1 -- 172.21.15.84:0/2021540858 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1076285554 0 0) 0x7fe9f41d39e0 con 0x7fe9f40b4200 2024-03-29T10:37:13.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.838+0000 7fe9f0ff9640 1 -- 172.21.15.84:0/2021540858 >> v1:172.21.15.84:6790/0 conn(0x7fe9f413f9a0 legacy=0x7fe9f41d3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:13.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.838+0000 7fe9f0ff9640 1 -- 172.21.15.84:0/2021540858 >> v1:172.21.15.86:6789/0 conn(0x7fe9f40b4920 legacy=0x7fe9f40a9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:13.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.838+0000 7fe9f0ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:37:13.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.838+0000 7fe9f0ff9640 1 monclient: found mon.a 2024-03-29T10:37:13.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.838+0000 7fe9f0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:13.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.838+0000 7fe9f0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:37:13.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.838+0000 7fe9f0ff9640 1 -- 172.21.15.84:0/2021540858 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f406f270 con 0x7fe9f40b4200 2024-03-29T10:37:13.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.838+0000 7fe9f0ff9640 10 monclient: _finish_auth 0 2024-03-29T10:37:13.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.838+0000 7fe9f0ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:37:13.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.838+0000 7fe9f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:13.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.838+0000 7fe9f9699640 5 monclient: authenticate success, global_id 4320 2024-03-29T10:37:13.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.838+0000 7fe9f9699640 10 monclient: _renew_subs 2024-03-29T10:37:13.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.838+0000 7fe9f9699640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:37:13.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.838+0000 7fe9f9699640 1 -- 172.21.15.84:0/2021540858 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9f406f520 con 0x7fe9f40b4200 2024-03-29T10:37:13.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.838+0000 7fe9f9699640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:37:13.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.838+0000 7fe9f9699640 10 monclient: _renew_subs 2024-03-29T10:37:13.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.838+0000 7fe9f9699640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:37:13.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.838+0000 7fe9f9699640 1 -- 172.21.15.84:0/2021540858 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9f40ade00 con 0x7fe9f40b4200 2024-03-29T10:37:13.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.838+0000 7fe9f9699640 1 librados: init done 2024-03-29T10:37:13.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.838+0000 7fe9f0ff9640 1 -- 172.21.15.84:0/2021540858 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9e4001cd0 con 0x7fe9f40b4200 2024-03-29T10:37:13.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.838+0000 7fe9f0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:13.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.838+0000 7fe9f0ff9640 1 -- 172.21.15.84:0/2021540858 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fe9e4003a60 con 0x7fe9f40b4200 2024-03-29T10:37:13.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.839+0000 7fe9f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:13.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.839+0000 7fe9f0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:37:13.830 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.839+0000 7fe9f0ff9640 10 monclient: dump: 2024-03-29T10:37:13.831 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:13.831 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:13.831 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:13.831 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:13.831 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:13.831 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:13.831 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:13.831 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:13.831 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:13.831 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:13.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.839+0000 7fe9f27fc640 4 set_mon_vals no callback set 2024-03-29T10:37:13.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.840+0000 7fe9f9699640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:13.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.840+0000 7fe9f9699640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:13.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.840+0000 7fe9f9699640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:37:13.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.840+0000 7fe9f9699640 1 -- 172.21.15.84:0/2021540858 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9f40ab520 con 0x7fe9f40b4200 2024-03-29T10:37:13.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.842+0000 7fe9f0ff9640 1 -- 172.21.15.84:0/2021540858 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7fe9e4003bc0 con 0x7fe9f40b4200 2024-03-29T10:37:13.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.843+0000 7fe9f37fe640 10 monclient: get_auth_request con 0x7fe9e006ddd0 auth_method 0 2024-03-29T10:37:13.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.843+0000 7fe9f0ff9640 1 -- 172.21.15.84:0/2021540858 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7fe9e4084ab0 con 0x7fe9f40b4200 2024-03-29T10:37:13.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.843+0000 7fe9f0ff9640 10 client.4320.objecter ms_dispatch 0x7fe9f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:37:13.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.843+0000 7fe9f0ff9640 3 client.4320.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:37:13.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.843+0000 7fe9f0ff9640 3 client.4320.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:37:13.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.843+0000 7fe9f0ff9640 20 client.4320.objecter dump_active .. 0 homeless 2024-03-29T10:37:13.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.844+0000 7fe9f0ff9640 10 client.4320.objecter ms_handle_connect 0x7fe9e006ddd0 2024-03-29T10:37:13.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.850+0000 7fe9f0ff9640 1 -- 172.21.15.84:0/2021540858 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9e4050390 con 0x7fe9f40b4200 2024-03-29T10:37:13.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.850+0000 7fe9f0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:13.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:13.850+0000 7fe9f0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:37:14.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:14.229+0000 7fe9f9699640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:14.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:14.229+0000 7fe9f9699640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:14.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:14.229+0000 7fe9f9699640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:37:14.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:14.229+0000 7fe9f9699640 1 -- 172.21.15.84:0/2021540858 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9f41d2a80 con 0x7fe9f40b4200 2024-03-29T10:37:14.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:14.230+0000 7fe9f0ff9640 1 -- 172.21.15.84:0/2021540858 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7fe9f41d2a80 con 0x7fe9f40b4200 2024-03-29T10:37:14.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:14.230+0000 7fe9f0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:14.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:14.230+0000 7fe9f0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:37:14.215 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:37:14.215 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:37:14.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:14.233+0000 7fe9f9699640 10 librados: watch_flush enter 2024-03-29T10:37:14.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:14.233+0000 7fe9f9699640 10 librados: watch_flush exit 2024-03-29T10:37:14.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:14.233+0000 7fe9f9699640 20 client.4320.objecter shutdown clearing up homeless session... 2024-03-29T10:37:14.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:14.233+0000 7fe9f9699640 10 client.4320.objecter successfully canceled tick 2024-03-29T10:37:14.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:14.233+0000 7fe9f9699640 1 -- 172.21.15.84:0/2021540858 >> v1:172.21.15.86:6816/3337225337 conn(0x7fe9e006ddd0 legacy=0x7fe9e0070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:14.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:14.233+0000 7fe9f9699640 10 monclient: shutdown 2024-03-29T10:37:14.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:14.233+0000 7fe9f9699640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:14.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:14.233+0000 7fe9f9699640 1 -- 172.21.15.84:0/2021540858 >> v1:172.21.15.84:6789/0 conn(0x7fe9f40b4200 legacy=0x7fe9f4071b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:14.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:14.234+0000 7fe9f9699640 1 -- 172.21.15.84:0/2021540858 shutdown_connections 2024-03-29T10:37:14.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:14.234+0000 7fe9f9699640 1 -- 172.21.15.84:0/2021540858 >> 172.21.15.84:0/2021540858 conn(0x7fe9f406d230 msgr2=0x7fe9f40b6bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:14.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:14.234+0000 7fe9f9699640 1 -- 172.21.15.84:0/2021540858 shutdown_connections 2024-03-29T10:37:14.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:14.234+0000 7fe9f9699640 1 -- 172.21.15.84:0/2021540858 wait complete. 2024-03-29T10:37:14.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:14.234+0000 7fe9f9699640 1 librados: shutdown 2024-03-29T10:37:14.233 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:37:21.040 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:37:21.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.184+0000 7f8549ae4640 10 monclient: get_monmap_and_config 2024-03-29T10:37:21.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.184+0000 7f8549ae4640 10 monclient: build_initial_monmap 2024-03-29T10:37:21.169 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.184+0000 7f8549ae4640 10 monclient: monmap: 2024-03-29T10:37:21.169 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:37:21.169 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:21.169 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:37:21.185882+0000 2024-03-29T10:37:21.170 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:37:21.185882+0000 2024-03-29T10:37:21.170 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:21.170 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:21.170 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:37:21.170 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:37:21.170 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:37:21.170 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:21.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.185+0000 7f8549ae4640 1 Processor -- start 2024-03-29T10:37:21.170 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.185+0000 7f8549ae4640 1 -- start start 2024-03-29T10:37:21.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.185+0000 7f8549ae4640 10 monclient: init 2024-03-29T10:37:21.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.186+0000 7f8549ae4640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:21.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.186+0000 7f8549ae4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:37:21.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.186+0000 7f8549ae4640 10 monclient(hunting): picked mon.noname-c con 0x7f85441422b0 addr v1:172.21.15.86:6789/0 2024-03-29T10:37:21.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.186+0000 7f8549ae4640 10 monclient(hunting): picked mon.noname-a con 0x7f8544142690 addr v1:172.21.15.84:6789/0 2024-03-29T10:37:21.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.186+0000 7f8549ae4640 10 monclient(hunting): picked mon.noname-b con 0x7f85440b17d0 addr v1:172.21.15.84:6790/0 2024-03-29T10:37:21.171 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.186+0000 7f8549ae4640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8544142c80 con 0x7f8544142690 2024-03-29T10:37:21.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.186+0000 7f8549ae4640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85440aae40 con 0x7f85441422b0 2024-03-29T10:37:21.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.186+0000 7f8549ae4640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f854413fbd0 con 0x7f85440b17d0 2024-03-29T10:37:21.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.186+0000 7f8549ae4640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:21.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.186+0000 7f8549ae4640 10 monclient(hunting): authenticate will time out at 943.449969s 2024-03-29T10:37:21.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.186+0000 7f8543fff640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f85440b17d0 0x7f85440b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:38836/0 (socket says 172.21.15.84:38836) 2024-03-29T10:37:21.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.186+0000 7f8543fff640 1 -- 172.21.15.84:0/2258085283 learned_addr learned my addr 172.21.15.84:0/2258085283 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:37:21.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.187+0000 7f85427fc640 1 -- 172.21.15.84:0/2258085283 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366695866 0 0) 0x7f85440aae40 con 0x7f85441422b0 2024-03-29T10:37:21.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.187+0000 7f85427fc640 10 monclient(hunting): _negotiate 2024-03-29T10:37:21.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.187+0000 7f85427fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:21.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.187+0000 7f85427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:21.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.187+0000 7f85427fc640 10 monclient(hunting): my global_id is 4309 2024-03-29T10:37:21.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.187+0000 7f85427fc640 1 -- 172.21.15.84:0/2258085283 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f852c003520 con 0x7f85441422b0 2024-03-29T10:37:21.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.187+0000 7f85427fc640 1 -- 172.21.15.84:0/2258085283 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3740661216 0 0) 0x7f854413fbd0 con 0x7f85440b17d0 2024-03-29T10:37:21.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.187+0000 7f85427fc640 10 monclient(hunting): _negotiate 2024-03-29T10:37:21.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.187+0000 7f85427fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:21.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.187+0000 7f85427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:21.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.187+0000 7f85427fc640 10 monclient(hunting): my global_id is 4337 2024-03-29T10:37:21.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.187+0000 7f85427fc640 1 -- 172.21.15.84:0/2258085283 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85440aae40 con 0x7f85440b17d0 2024-03-29T10:37:21.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.188+0000 7f85427fc640 1 -- 172.21.15.84:0/2258085283 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1138105194 0 0) 0x7f85440aae40 con 0x7f85440b17d0 2024-03-29T10:37:21.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.188+0000 7f85427fc640 1 -- 172.21.15.84:0/2258085283 >> v1:172.21.15.86:6789/0 conn(0x7f85441422b0 legacy=0x7f854413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:21.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.188+0000 7f85427fc640 1 -- 172.21.15.84:0/2258085283 >> v1:172.21.15.84:6789/0 conn(0x7f8544142690 legacy=0x7f85440b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:37:21.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.189+0000 7f85427fc640 10 monclient: _finish_hunting 0 2024-03-29T10:37:21.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.189+0000 7f85427fc640 1 monclient: found mon.noname-b 2024-03-29T10:37:21.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.190+0000 7f85427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:21.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.190+0000 7f85427fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:37:21.175 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.190+0000 7f85427fc640 1 -- 172.21.15.84:0/2258085283 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8544151940 con 0x7f85440b17d0 2024-03-29T10:37:21.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.190+0000 7f85427fc640 10 monclient: _finish_auth 0 2024-03-29T10:37:21.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.190+0000 7f85427fc640 10 monclient: _check_auth_tickets 2024-03-29T10:37:21.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.190+0000 7f85427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:21.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.190+0000 7f85427fc640 1 -- 172.21.15.84:0/2258085283 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f853c0019f0 con 0x7f85440b17d0 2024-03-29T10:37:21.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.190+0000 7f85427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:21.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.190+0000 7f85427fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:37:21.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.190+0000 7f85427fc640 10 monclient: dump: 2024-03-29T10:37:21.176 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:21.176 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:21.176 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:21.176 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:21.176 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:21.176 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:21.177 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:21.177 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:21.177 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:21.177 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:21.177 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.190+0000 7f8549ae4640 5 monclient: authenticate success, global_id 4337 2024-03-29T10:37:21.177 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.190+0000 7f8549ae4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:37:21.177 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.190+0000 7f85427fc640 1 -- 172.21.15.84:0/2258085283 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f853c001b50 con 0x7f85440b17d0 2024-03-29T10:37:21.177 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.190+0000 7f85427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:21.177 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.190+0000 7f85427fc640 1 -- 172.21.15.84:0/2258085283 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f853c001ee0 con 0x7f85440b17d0 2024-03-29T10:37:21.177 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.190+0000 7f85427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:21.177 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.190+0000 7f85427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:37:21.177 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.190+0000 7f85427fc640 10 monclient: dump: 2024-03-29T10:37:21.177 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:21.177 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:21.178 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:21.178 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:21.178 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:21.178 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:21.178 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:21.178 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:21.178 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:21.178 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:21.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.191+0000 7f8549ae4640 10 monclient: get_monmap_and_config success 2024-03-29T10:37:21.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.191+0000 7f8549ae4640 4 set_mon_vals no callback set 2024-03-29T10:37:21.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.191+0000 7f8549ae4640 10 monclient: shutdown 2024-03-29T10:37:21.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.191+0000 7f8549ae4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:21.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.191+0000 7f8549ae4640 1 -- 172.21.15.84:0/2258085283 >> v1:172.21.15.84:6790/0 conn(0x7f85440b17d0 legacy=0x7f85440b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:21.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.192+0000 7f8549ae4640 1 -- 172.21.15.84:0/2258085283 shutdown_connections 2024-03-29T10:37:21.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.192+0000 7f8549ae4640 1 -- 172.21.15.84:0/2258085283 >> 172.21.15.84:0/2258085283 conn(0x7f8544066a00 msgr2=0x7f8544066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:21.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.192+0000 7f8549ae4640 1 -- 172.21.15.84:0/2258085283 shutdown_connections 2024-03-29T10:37:21.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.192+0000 7f8549ae4640 1 -- 172.21.15.84:0/2258085283 wait complete. 2024-03-29T10:37:21.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.193+0000 7f8549ae4640 10 monclient: build_initial_monmap 2024-03-29T10:37:21.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.193+0000 7f8549ae4640 10 monclient: monmap: 2024-03-29T10:37:21.186 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:37:21.186 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:37:21.186 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:37:21.186 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:37:21.186 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:21.186 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:21.186 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:37:21.186 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:37:21.186 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:37:21.187 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:21.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.193+0000 7f8549ae4640 1 librados: starting msgr at 2024-03-29T10:37:21.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.193+0000 7f8549ae4640 1 librados: starting objecter 2024-03-29T10:37:21.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.193+0000 7f8549ae4640 1 Processor -- start 2024-03-29T10:37:21.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.193+0000 7f8549ae4640 1 -- start start 2024-03-29T10:37:21.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.193+0000 7f8549ae4640 1 librados: setting wanted keys 2024-03-29T10:37:21.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.193+0000 7f8549ae4640 1 librados: calling monclient init 2024-03-29T10:37:21.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.193+0000 7f8549ae4640 10 monclient: init 2024-03-29T10:37:21.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.194+0000 7f8549ae4640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:21.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.194+0000 7f8549ae4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:37:21.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.194+0000 7f8549ae4640 10 monclient(hunting): picked mon.noname-a con 0x7f85440b17d0 addr v1:172.21.15.84:6789/0 2024-03-29T10:37:21.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.195+0000 7f8549ae4640 10 monclient(hunting): picked mon.noname-c con 0x7f85441422b0 addr v1:172.21.15.84:6790/0 2024-03-29T10:37:21.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.195+0000 7f8549ae4640 10 monclient(hunting): picked mon.noname-b con 0x7f8544142690 addr v1:172.21.15.86:6789/0 2024-03-29T10:37:21.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.195+0000 7f8549ae4640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85440ac6c0 con 0x7f85440b17d0 2024-03-29T10:37:21.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.195+0000 7f8549ae4640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85441db6b0 con 0x7f8544142690 2024-03-29T10:37:21.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.195+0000 7f8549ae4640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85440ac930 con 0x7f85441422b0 2024-03-29T10:37:21.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.195+0000 7f8549ae4640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:21.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.195+0000 7f8549ae4640 10 monclient(hunting): authenticate will time out at 943.458540s 2024-03-29T10:37:21.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.195+0000 7f8542ffd640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f85441422b0 0x7f85441da890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:39310/0 (socket says 172.21.15.84:39310) 2024-03-29T10:37:21.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.195+0000 7f8542ffd640 1 -- 172.21.15.84:0/2682259833 learned_addr learned my addr 172.21.15.84:0/2682259833 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:37:21.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.195+0000 7f852bfff640 10 client.?.objecter ms_handle_connect 0x7f85441422b0 2024-03-29T10:37:21.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.195+0000 7f852bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:21.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.195+0000 7f852bfff640 1 -- 172.21.15.84:0/2682259833 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3737784422 0 0) 0x7f85440ac930 con 0x7f85441422b0 2024-03-29T10:37:21.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.195+0000 7f852bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:37:21.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.196+0000 7f852bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:21.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.196+0000 7f852bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:21.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.196+0000 7f852bfff640 10 monclient(hunting): my global_id is 4340 2024-03-29T10:37:21.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.196+0000 7f852bfff640 1 -- 172.21.15.84:0/2682259833 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8530004eb0 con 0x7f85441422b0 2024-03-29T10:37:21.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.196+0000 7f852bfff640 1 -- 172.21.15.84:0/2682259833 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3947736987 0 0) 0x7f8530004eb0 con 0x7f85441422b0 2024-03-29T10:37:21.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.196+0000 7f852bfff640 1 -- 172.21.15.84:0/2682259833 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85440ac930 con 0x7f85441422b0 2024-03-29T10:37:21.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.196+0000 7f852bfff640 10 client.?.objecter ms_handle_connect 0x7f8544142690 2024-03-29T10:37:21.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.196+0000 7f852bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:21.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.196+0000 7f852bfff640 1 -- 172.21.15.84:0/2682259833 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f8538003fe0 con 0x7f85441422b0 2024-03-29T10:37:21.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.196+0000 7f852bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:21.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.196+0000 7f852bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:37:21.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.197+0000 7f852bfff640 10 monclient(hunting): dump: 2024-03-29T10:37:21.189 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:21.189 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:21.189 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:21.189 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:21.189 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:21.189 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:21.189 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:21.190 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:21.190 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:21.190 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:21.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.197+0000 7f852bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:37:21.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.197+0000 7f852bfff640 1 -- 172.21.15.84:0/2682259833 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3422867257 0 0) 0x7f85440ac930 con 0x7f85441422b0 2024-03-29T10:37:21.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.197+0000 7f852bfff640 1 -- 172.21.15.84:0/2682259833 >> v1:172.21.15.86:6789/0 conn(0x7f8544142690 legacy=0x7f85441dafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:21.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.197+0000 7f852bfff640 1 -- 172.21.15.84:0/2682259833 >> v1:172.21.15.84:6789/0 conn(0x7f85440b17d0 legacy=0x7f854412dda0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:37:21.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.197+0000 7f852bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:37:21.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.197+0000 7f852bfff640 1 monclient: found mon.c 2024-03-29T10:37:21.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.197+0000 7f852bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:21.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.197+0000 7f852bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:37:21.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.197+0000 7f852bfff640 1 -- 172.21.15.84:0/2682259833 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85440acc10 con 0x7f85441422b0 2024-03-29T10:37:21.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.197+0000 7f852bfff640 10 monclient: _finish_auth 0 2024-03-29T10:37:21.191 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.197+0000 7f852bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:37:21.191 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.197+0000 7f852bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:21.191 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.197+0000 7f8549ae4640 5 monclient: authenticate success, global_id 4340 2024-03-29T10:37:21.191 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.197+0000 7f8549ae4640 10 monclient: _renew_subs 2024-03-29T10:37:21.191 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.197+0000 7f8549ae4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:37:21.191 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.197+0000 7f8549ae4640 1 -- 172.21.15.84:0/2682259833 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8544131bf0 con 0x7f85441422b0 2024-03-29T10:37:21.191 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.197+0000 7f8549ae4640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:37:21.191 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.197+0000 7f8549ae4640 10 monclient: _renew_subs 2024-03-29T10:37:21.191 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.197+0000 7f8549ae4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:37:21.191 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.197+0000 7f8549ae4640 1 -- 172.21.15.84:0/2682259833 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f854412e4b0 con 0x7f85441422b0 2024-03-29T10:37:21.191 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.197+0000 7f8549ae4640 1 librados: init done 2024-03-29T10:37:21.191 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.198+0000 7f852bfff640 1 -- 172.21.15.84:0/2682259833 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8538004240 con 0x7f85441422b0 2024-03-29T10:37:21.191 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.198+0000 7f852bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:21.191 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.198+0000 7f8541ffb640 4 set_mon_vals no callback set 2024-03-29T10:37:21.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.200+0000 7f852bfff640 1 -- 172.21.15.84:0/2682259833 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f8538001320 con 0x7f85441422b0 2024-03-29T10:37:21.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.200+0000 7f852bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:21.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.200+0000 7f852bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:37:21.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.200+0000 7f852bfff640 10 monclient: dump: 2024-03-29T10:37:21.192 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:21.192 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:21.192 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:21.192 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:21.192 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:21.192 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:21.192 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:21.192 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:21.192 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:21.192 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:21.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.200+0000 7f852bfff640 1 -- 172.21.15.84:0/2682259833 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f8538001540 con 0x7f85441422b0 2024-03-29T10:37:21.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.200+0000 7f8549ae4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:21.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.200+0000 7f8549ae4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:21.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.200+0000 7f8549ae4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:37:21.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.200+0000 7f8549ae4640 1 -- 172.21.15.84:0/2682259833 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85440ab720 con 0x7f85441422b0 2024-03-29T10:37:21.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.201+0000 7f852bfff640 1 -- 172.21.15.84:0/2682259833 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f8538084b20 con 0x7f85441422b0 2024-03-29T10:37:21.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.201+0000 7f852bfff640 10 client.4340.objecter ms_dispatch 0x7f8544000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:37:21.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.201+0000 7f852bfff640 3 client.4340.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:37:21.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.201+0000 7f852bfff640 3 client.4340.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:37:21.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.201+0000 7f85437fe640 10 monclient: get_auth_request con 0x7f853006dbd0 auth_method 0 2024-03-29T10:37:21.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.201+0000 7f852bfff640 20 client.4340.objecter dump_active .. 0 homeless 2024-03-29T10:37:21.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.201+0000 7f852bfff640 10 client.4340.objecter ms_handle_connect 0x7f853006dbd0 2024-03-29T10:37:21.194 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.207+0000 7f852bfff640 1 -- 172.21.15.84:0/2682259833 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8538050480 con 0x7f85441422b0 2024-03-29T10:37:21.194 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.207+0000 7f852bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:21.194 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.207+0000 7f852bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:37:21.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.524+0000 7f8549ae4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:21.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.524+0000 7f8549ae4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:21.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.524+0000 7f8549ae4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:37:21.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.524+0000 7f8549ae4640 1 -- 172.21.15.84:0/2682259833 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85440b6d10 con 0x7f85441422b0 2024-03-29T10:37:21.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.526+0000 7f852bfff640 1 -- 172.21.15.84:0/2682259833 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f85440b6d10 con 0x7f85441422b0 2024-03-29T10:37:21.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.526+0000 7f852bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:21.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.526+0000 7f852bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:37:21.511 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:37:21.512 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:37:21.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.530+0000 7f8549ae4640 10 librados: watch_flush enter 2024-03-29T10:37:21.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.530+0000 7f8549ae4640 10 librados: watch_flush exit 2024-03-29T10:37:21.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.530+0000 7f8549ae4640 20 client.4340.objecter shutdown clearing up homeless session... 2024-03-29T10:37:21.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.530+0000 7f8549ae4640 10 client.4340.objecter successfully canceled tick 2024-03-29T10:37:21.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.530+0000 7f8549ae4640 1 -- 172.21.15.84:0/2682259833 >> v1:172.21.15.86:6816/3337225337 conn(0x7f853006dbd0 legacy=0x7f8530070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:21.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.530+0000 7f8549ae4640 10 monclient: shutdown 2024-03-29T10:37:21.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.530+0000 7f8549ae4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:21.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.530+0000 7f8549ae4640 1 -- 172.21.15.84:0/2682259833 >> v1:172.21.15.84:6790/0 conn(0x7f85441422b0 legacy=0x7f85441da890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:21.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.530+0000 7f8549ae4640 1 -- 172.21.15.84:0/2682259833 shutdown_connections 2024-03-29T10:37:21.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.530+0000 7f8549ae4640 1 -- 172.21.15.84:0/2682259833 >> 172.21.15.84:0/2682259833 conn(0x7f8544066a00 msgr2=0x7f854413c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:21.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.531+0000 7f8549ae4640 1 -- 172.21.15.84:0/2682259833 shutdown_connections 2024-03-29T10:37:21.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.531+0000 7f8549ae4640 1 -- 172.21.15.84:0/2682259833 wait complete. 2024-03-29T10:37:21.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:21.531+0000 7f8549ae4640 1 librados: shutdown 2024-03-29T10:37:21.532 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:37:28.341 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:37:28.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.513+0000 7f980460f640 10 monclient: get_monmap_and_config 2024-03-29T10:37:28.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.513+0000 7f980460f640 10 monclient: build_initial_monmap 2024-03-29T10:37:28.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.513+0000 7f980460f640 10 monclient: monmap: 2024-03-29T10:37:28.498 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:37:28.498 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:28.498 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:37:28.514144+0000 2024-03-29T10:37:28.498 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:37:28.514144+0000 2024-03-29T10:37:28.498 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:28.498 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:28.498 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:37:28.498 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:37:28.499 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:37:28.499 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:28.500 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.516+0000 7f980460f640 1 Processor -- start 2024-03-29T10:37:28.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.516+0000 7f980460f640 1 -- start start 2024-03-29T10:37:28.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.516+0000 7f980460f640 10 monclient: init 2024-03-29T10:37:28.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.517+0000 7f980460f640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:28.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.517+0000 7f980460f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:37:28.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.517+0000 7f980460f640 10 monclient(hunting): picked mon.noname-c con 0x7f97fc1422b0 addr v1:172.21.15.86:6789/0 2024-03-29T10:37:28.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.518+0000 7f980460f640 10 monclient(hunting): picked mon.noname-a con 0x7f97fc142690 addr v1:172.21.15.84:6789/0 2024-03-29T10:37:28.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.518+0000 7f980460f640 10 monclient(hunting): picked mon.noname-b con 0x7f97fc0b17d0 addr v1:172.21.15.84:6790/0 2024-03-29T10:37:28.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.518+0000 7f980460f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97fc142c80 con 0x7f97fc142690 2024-03-29T10:37:28.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.518+0000 7f980460f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97fc0aae40 con 0x7f97fc1422b0 2024-03-29T10:37:28.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.518+0000 7f980460f640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97fc13fbd0 con 0x7f97fc0b17d0 2024-03-29T10:37:28.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.518+0000 7f980460f640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:28.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.518+0000 7f980460f640 10 monclient(hunting): authenticate will time out at 950.781385s 2024-03-29T10:37:28.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.521+0000 7f9801baa640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f97fc142690 0x7f97fc0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:42769/0 (socket says 172.21.15.84:42769) 2024-03-29T10:37:28.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.521+0000 7f9801baa640 1 -- 172.21.15.84:0/3249533327 learned_addr learned my addr 172.21.15.84:0/3249533327 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:37:28.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.522+0000 7f98013a9640 1 -- 172.21.15.84:0/3249533327 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268196877 0 0) 0x7f97fc142c80 con 0x7f97fc142690 2024-03-29T10:37:28.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.522+0000 7f98013a9640 10 monclient(hunting): _negotiate 2024-03-29T10:37:28.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.522+0000 7f98013a9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:28.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.522+0000 7f98013a9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:28.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.522+0000 7f98013a9640 10 monclient(hunting): my global_id is 4323 2024-03-29T10:37:28.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.522+0000 7f98013a9640 1 -- 172.21.15.84:0/3249533327 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97e8003520 con 0x7f97fc142690 2024-03-29T10:37:28.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.522+0000 7f98013a9640 1 -- 172.21.15.84:0/3249533327 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 538018698 0 0) 0x7f97e8003520 con 0x7f97fc142690 2024-03-29T10:37:28.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.522+0000 7f98013a9640 1 -- 172.21.15.84:0/3249533327 >> v1:172.21.15.84:6790/0 conn(0x7f97fc0b17d0 legacy=0x7f97fc0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:28.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.522+0000 7f98013a9640 1 -- 172.21.15.84:0/3249533327 >> v1:172.21.15.86:6789/0 conn(0x7f97fc1422b0 legacy=0x7f97fc13f4c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:37:28.510 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.522+0000 7f98013a9640 10 monclient: _finish_hunting 0 2024-03-29T10:37:28.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.522+0000 7f98013a9640 1 monclient: found mon.noname-a 2024-03-29T10:37:28.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.522+0000 7f98013a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:28.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.522+0000 7f98013a9640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T10:37:28.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.522+0000 7f98013a9640 1 -- 172.21.15.84:0/3249533327 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97fc151940 con 0x7f97fc142690 2024-03-29T10:37:28.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.522+0000 7f98013a9640 10 monclient: _finish_auth 0 2024-03-29T10:37:28.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.522+0000 7f98013a9640 10 monclient: _check_auth_tickets 2024-03-29T10:37:28.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.522+0000 7f98013a9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:28.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.522+0000 7f98013a9640 1 -- 172.21.15.84:0/3249533327 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f97ec001690 con 0x7f97fc142690 2024-03-29T10:37:28.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.522+0000 7f98013a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:28.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.522+0000 7f98013a9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:37:28.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.522+0000 7f98013a9640 10 monclient: dump: 2024-03-29T10:37:28.511 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:28.511 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:28.511 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:28.512 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:28.512 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:28.512 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:28.512 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:28.512 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:28.512 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:28.512 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:28.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.522+0000 7f98013a9640 1 -- 172.21.15.84:0/3249533327 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97ec0018f0 con 0x7f97fc142690 2024-03-29T10:37:28.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.522+0000 7f98013a9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:28.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.522+0000 7f98013a9640 1 -- 172.21.15.84:0/3249533327 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f97ec001c80 con 0x7f97fc142690 2024-03-29T10:37:28.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.522+0000 7f98013a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:28.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.522+0000 7f98013a9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:37:28.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.522+0000 7f98013a9640 10 monclient: dump: 2024-03-29T10:37:28.512 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:28.513 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:28.513 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:28.513 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:28.513 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:28.513 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:28.513 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:28.513 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:28.513 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:28.513 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:28.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.523+0000 7f980460f640 5 monclient: authenticate success, global_id 4323 2024-03-29T10:37:28.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.523+0000 7f980460f640 10 monclient: get_monmap_and_config success 2024-03-29T10:37:28.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.523+0000 7f980460f640 4 set_mon_vals no callback set 2024-03-29T10:37:28.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.524+0000 7f980460f640 10 monclient: shutdown 2024-03-29T10:37:28.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.524+0000 7f980460f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:28.513 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.524+0000 7f980460f640 1 -- 172.21.15.84:0/3249533327 >> v1:172.21.15.84:6789/0 conn(0x7f97fc142690 legacy=0x7f97fc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:28.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.524+0000 7f980460f640 1 -- 172.21.15.84:0/3249533327 shutdown_connections 2024-03-29T10:37:28.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.524+0000 7f980460f640 1 -- 172.21.15.84:0/3249533327 >> 172.21.15.84:0/3249533327 conn(0x7f97fc066a00 msgr2=0x7f97fc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:28.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.525+0000 7f980460f640 1 -- 172.21.15.84:0/3249533327 shutdown_connections 2024-03-29T10:37:28.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.525+0000 7f980460f640 1 -- 172.21.15.84:0/3249533327 wait complete. 2024-03-29T10:37:28.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.526+0000 7f980460f640 10 monclient: build_initial_monmap 2024-03-29T10:37:28.515 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.526+0000 7f980460f640 10 monclient: monmap: 2024-03-29T10:37:28.515 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:37:28.515 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:37:28.515 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:37:28.515 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:37:28.515 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:28.515 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:28.515 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:37:28.515 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:37:28.515 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:37:28.516 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:28.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.526+0000 7f980460f640 1 librados: starting msgr at 2024-03-29T10:37:28.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.526+0000 7f980460f640 1 librados: starting objecter 2024-03-29T10:37:28.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.526+0000 7f980460f640 1 Processor -- start 2024-03-29T10:37:28.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.526+0000 7f980460f640 1 -- start start 2024-03-29T10:37:28.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.526+0000 7f980460f640 1 librados: setting wanted keys 2024-03-29T10:37:28.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.526+0000 7f980460f640 1 librados: calling monclient init 2024-03-29T10:37:28.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.526+0000 7f980460f640 10 monclient: init 2024-03-29T10:37:28.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.527+0000 7f980460f640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:28.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.527+0000 7f980460f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:37:28.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.527+0000 7f980460f640 10 monclient(hunting): picked mon.noname-c con 0x7f97fc0b17d0 addr v1:172.21.15.84:6790/0 2024-03-29T10:37:28.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.529+0000 7f980460f640 10 monclient(hunting): picked mon.noname-b con 0x7f97fc1422b0 addr v1:172.21.15.86:6789/0 2024-03-29T10:37:28.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.529+0000 7f980460f640 10 monclient(hunting): picked mon.noname-a con 0x7f97fc142690 addr v1:172.21.15.84:6789/0 2024-03-29T10:37:28.516 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.529+0000 7f980460f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97fc12d660 con 0x7f97fc142690 2024-03-29T10:37:28.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.529+0000 7f980460f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97fc0ac0a0 con 0x7f97fc1422b0 2024-03-29T10:37:28.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.529+0000 7f980460f640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97fc0ac380 con 0x7f97fc0b17d0 2024-03-29T10:37:28.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.529+0000 7f980460f640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:28.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.529+0000 7f980460f640 10 monclient(hunting): authenticate will time out at 950.792311s 2024-03-29T10:37:28.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.529+0000 7f98023ab640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f97fc0b17d0 0x7f97fc1323f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:46538/0 (socket says 172.21.15.84:46538) 2024-03-29T10:37:28.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.529+0000 7f98023ab640 1 -- 172.21.15.84:0/3812201079 learned_addr learned my addr 172.21.15.84:0/3812201079 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:37:28.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.529+0000 7f97f2ffd640 10 client.?.objecter ms_handle_connect 0x7f97fc0b17d0 2024-03-29T10:37:28.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.529+0000 7f97f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:28.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.530+0000 7f97f2ffd640 1 -- 172.21.15.84:0/3812201079 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464729219 0 0) 0x7f97fc0ac380 con 0x7f97fc0b17d0 2024-03-29T10:37:28.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.530+0000 7f97f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:37:28.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.530+0000 7f97f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:28.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.530+0000 7f97f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:28.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.530+0000 7f97f2ffd640 10 monclient(hunting): my global_id is 4346 2024-03-29T10:37:28.517 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.530+0000 7f97f2ffd640 1 -- 172.21.15.84:0/3812201079 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97e0004eb0 con 0x7f97fc0b17d0 2024-03-29T10:37:28.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.531+0000 7f97f2ffd640 1 -- 172.21.15.84:0/3812201079 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1135122708 0 0) 0x7f97e0004eb0 con 0x7f97fc0b17d0 2024-03-29T10:37:28.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.531+0000 7f97f2ffd640 1 -- 172.21.15.84:0/3812201079 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97fc0ac380 con 0x7f97fc0b17d0 2024-03-29T10:37:28.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.531+0000 7f97f2ffd640 1 -- 172.21.15.84:0/3812201079 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f97f8001870 con 0x7f97fc0b17d0 2024-03-29T10:37:28.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.531+0000 7f97f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:28.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.531+0000 7f97f2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:37:28.518 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.531+0000 7f97f2ffd640 10 monclient(hunting): dump: 2024-03-29T10:37:28.518 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:28.518 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:28.518 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:28.518 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:28.518 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:28.518 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:28.518 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:28.519 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:28.519 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:28.519 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:28.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.531+0000 7f97f2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:37:28.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.531+0000 7f97f2ffd640 1 -- 172.21.15.84:0/3812201079 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2038038081 0 0) 0x7f97fc0ac380 con 0x7f97fc0b17d0 2024-03-29T10:37:28.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.531+0000 7f97f2ffd640 1 -- 172.21.15.84:0/3812201079 >> v1:172.21.15.86:6789/0 conn(0x7f97fc1422b0 legacy=0x7f97fc12c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:28.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.531+0000 7f97f2ffd640 1 -- 172.21.15.84:0/3812201079 >> v1:172.21.15.84:6789/0 conn(0x7f97fc142690 legacy=0x7f97fc12cf50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:37:28.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.532+0000 7f97f2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:37:28.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.532+0000 7f97f2ffd640 1 monclient: found mon.c 2024-03-29T10:37:28.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.532+0000 7f97f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:28.522 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.532+0000 7f97f2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:37:28.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.532+0000 7f97f2ffd640 1 -- 172.21.15.84:0/3812201079 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97fc0b1470 con 0x7f97fc0b17d0 2024-03-29T10:37:28.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.532+0000 7f97f2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:37:28.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.532+0000 7f97f2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:37:28.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.532+0000 7f97f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:28.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.533+0000 7f980460f640 5 monclient: authenticate success, global_id 4346 2024-03-29T10:37:28.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.533+0000 7f980460f640 10 monclient: _renew_subs 2024-03-29T10:37:28.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.533+0000 7f980460f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:37:28.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.533+0000 7f980460f640 1 -- 172.21.15.84:0/3812201079 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97fc131e70 con 0x7f97fc0b17d0 2024-03-29T10:37:28.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.533+0000 7f980460f640 10 client.4346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:37:28.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.533+0000 7f980460f640 10 monclient: _renew_subs 2024-03-29T10:37:28.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.533+0000 7f980460f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:37:28.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.533+0000 7f980460f640 1 -- 172.21.15.84:0/3812201079 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f97fc12fd30 con 0x7f97fc0b17d0 2024-03-29T10:37:28.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.533+0000 7f980460f640 1 librados: init done 2024-03-29T10:37:28.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.533+0000 7f97f2ffd640 1 -- 172.21.15.84:0/3812201079 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97f80019d0 con 0x7f97fc0b17d0 2024-03-29T10:37:28.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.533+0000 7f97f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:28.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.533+0000 7f97f2ffd640 1 -- 172.21.15.84:0/3812201079 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f97f8004ae0 con 0x7f97fc0b17d0 2024-03-29T10:37:28.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.533+0000 7f97f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:28.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.533+0000 7f97f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:37:28.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.533+0000 7f97f2ffd640 10 monclient: dump: 2024-03-29T10:37:28.524 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:28.524 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:28.524 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:28.524 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:28.524 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:28.524 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:28.524 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:28.524 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:28.524 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:28.524 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:28.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.533+0000 7f9800ba8640 4 set_mon_vals no callback set 2024-03-29T10:37:28.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.536+0000 7f980460f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:28.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.536+0000 7f980460f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:28.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.536+0000 7f980460f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:37:28.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.536+0000 7f980460f640 1 -- 172.21.15.84:0/3812201079 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97fc12ffe0 con 0x7f97fc0b17d0 2024-03-29T10:37:28.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.536+0000 7f97f2ffd640 1 -- 172.21.15.84:0/3812201079 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f97f8004c40 con 0x7f97fc0b17d0 2024-03-29T10:37:28.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.536+0000 7f97f2ffd640 1 -- 172.21.15.84:0/3812201079 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f97f8084b60 con 0x7f97fc0b17d0 2024-03-29T10:37:28.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.536+0000 7f97f2ffd640 10 client.4346.objecter ms_dispatch 0x7f97fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:37:28.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.536+0000 7f97f2ffd640 3 client.4346.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:37:28.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.536+0000 7f97f2ffd640 3 client.4346.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:37:28.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.537+0000 7f97f2ffd640 20 client.4346.objecter dump_active .. 0 homeless 2024-03-29T10:37:28.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.537+0000 7f9801baa640 10 monclient: get_auth_request con 0x7f97e006dc80 auth_method 0 2024-03-29T10:37:28.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.537+0000 7f97f2ffd640 10 client.4346.objecter ms_handle_connect 0x7f97e006dc80 2024-03-29T10:37:28.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.547+0000 7f97f2ffd640 1 -- 172.21.15.84:0/3812201079 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f97fc12ffe0 con 0x7f97fc0b17d0 2024-03-29T10:37:28.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.547+0000 7f97f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:28.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.547+0000 7f97f2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:37:28.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.876+0000 7f980460f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:28.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.876+0000 7f980460f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:28.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.876+0000 7f980460f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:37:28.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.876+0000 7f980460f640 1 -- 172.21.15.84:0/3812201079 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97fc12ffe0 con 0x7f97fc0b17d0 2024-03-29T10:37:28.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.876+0000 7f97f2ffd640 1 -- 172.21.15.84:0/3812201079 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f97fc12ffe0 con 0x7f97fc0b17d0 2024-03-29T10:37:28.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.876+0000 7f97f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:28.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.876+0000 7f97f2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:37:28.862 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:37:28.862 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:37:28.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.879+0000 7f980460f640 10 librados: watch_flush enter 2024-03-29T10:37:28.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.880+0000 7f980460f640 10 librados: watch_flush exit 2024-03-29T10:37:28.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.880+0000 7f980460f640 20 client.4346.objecter shutdown clearing up homeless session... 2024-03-29T10:37:28.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.880+0000 7f980460f640 10 client.4346.objecter successfully canceled tick 2024-03-29T10:37:28.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.880+0000 7f980460f640 1 -- 172.21.15.84:0/3812201079 >> v1:172.21.15.86:6816/3337225337 conn(0x7f97e006dc80 legacy=0x7f97e0070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:28.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.880+0000 7f980460f640 10 monclient: shutdown 2024-03-29T10:37:28.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.880+0000 7f980460f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:28.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.880+0000 7f980460f640 1 -- 172.21.15.84:0/3812201079 >> v1:172.21.15.84:6790/0 conn(0x7f97fc0b17d0 legacy=0x7f97fc1323f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:28.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.880+0000 7f980460f640 1 -- 172.21.15.84:0/3812201079 shutdown_connections 2024-03-29T10:37:28.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.880+0000 7f980460f640 1 -- 172.21.15.84:0/3812201079 >> 172.21.15.84:0/3812201079 conn(0x7f97fc066a00 msgr2=0x7f97fc0b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:28.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.880+0000 7f980460f640 1 -- 172.21.15.84:0/3812201079 shutdown_connections 2024-03-29T10:37:28.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.880+0000 7f980460f640 1 -- 172.21.15.84:0/3812201079 wait complete. 2024-03-29T10:37:28.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:28.881+0000 7f980460f640 1 librados: shutdown 2024-03-29T10:37:28.882 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:37:35.689 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:37:35.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.850+0000 7fc535419640 10 monclient: get_monmap_and_config 2024-03-29T10:37:35.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.850+0000 7fc535419640 10 monclient: build_initial_monmap 2024-03-29T10:37:35.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.850+0000 7fc535419640 10 monclient: monmap: 2024-03-29T10:37:35.836 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:37:35.836 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:35.836 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:37:35.851456+0000 2024-03-29T10:37:35.836 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:37:35.851456+0000 2024-03-29T10:37:35.836 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:35.836 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:35.836 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:37:35.836 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:37:35.836 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:37:35.836 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:35.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.851+0000 7fc535419640 1 Processor -- start 2024-03-29T10:37:35.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.852+0000 7fc535419640 1 -- start start 2024-03-29T10:37:35.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.852+0000 7fc535419640 10 monclient: init 2024-03-29T10:37:35.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.852+0000 7fc535419640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:35.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.852+0000 7fc535419640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:37:35.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.852+0000 7fc535419640 10 monclient(hunting): picked mon.noname-b con 0x7fc5300a9750 addr v1:172.21.15.84:6790/0 2024-03-29T10:37:35.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.852+0000 7fc535419640 10 monclient(hunting): picked mon.noname-c con 0x7fc5300aad20 addr v1:172.21.15.86:6789/0 2024-03-29T10:37:35.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.852+0000 7fc535419640 10 monclient(hunting): picked mon.noname-a con 0x7fc5300a9b30 addr v1:172.21.15.84:6789/0 2024-03-29T10:37:35.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.852+0000 7fc535419640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5300af4d0 con 0x7fc5300a9b30 2024-03-29T10:37:35.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.852+0000 7fc535419640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5300af7b0 con 0x7fc5300aad20 2024-03-29T10:37:35.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.852+0000 7fc535419640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5300acab0 con 0x7fc5300a9750 2024-03-29T10:37:35.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.852+0000 7fc535419640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:35.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.852+0000 7fc535419640 10 monclient(hunting): authenticate will time out at 958.116053s 2024-03-29T10:37:35.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.853+0000 7fc52ffff640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fc5300a9750 0x7fc5300ac3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:40184/0 (socket says 172.21.15.84:40184) 2024-03-29T10:37:35.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.853+0000 7fc52ffff640 1 -- 172.21.15.84:0/3625144426 learned_addr learned my addr 172.21.15.84:0/3625144426 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:37:35.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.855+0000 7fc52effd640 1 -- 172.21.15.84:0/3625144426 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2463239987 0 0) 0x7fc5300acab0 con 0x7fc5300a9750 2024-03-29T10:37:35.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.855+0000 7fc52effd640 10 monclient(hunting): _negotiate 2024-03-29T10:37:35.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.855+0000 7fc52effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:35.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.855+0000 7fc52effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:35.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.855+0000 7fc52effd640 10 monclient(hunting): my global_id is 4349 2024-03-29T10:37:35.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.855+0000 7fc52effd640 1 -- 172.21.15.84:0/3625144426 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc518003520 con 0x7fc5300a9750 2024-03-29T10:37:35.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.856+0000 7fc52effd640 1 -- 172.21.15.84:0/3625144426 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022678501 0 0) 0x7fc5300af7b0 con 0x7fc5300aad20 2024-03-29T10:37:35.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.856+0000 7fc52effd640 10 monclient(hunting): _negotiate 2024-03-29T10:37:35.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.856+0000 7fc52effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:35.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.856+0000 7fc52effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:35.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.856+0000 7fc52effd640 10 monclient(hunting): my global_id is 4315 2024-03-29T10:37:35.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.856+0000 7fc52effd640 1 -- 172.21.15.84:0/3625144426 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5300acab0 con 0x7fc5300aad20 2024-03-29T10:37:35.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.856+0000 7fc52effd640 1 -- 172.21.15.84:0/3625144426 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1618406709 0 0) 0x7fc5300af4d0 con 0x7fc5300a9b30 2024-03-29T10:37:35.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.856+0000 7fc52effd640 10 monclient(hunting): _negotiate 2024-03-29T10:37:35.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.856+0000 7fc52effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:35.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.856+0000 7fc52effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:35.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.856+0000 7fc52effd640 10 monclient(hunting): my global_id is 4326 2024-03-29T10:37:35.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.856+0000 7fc52effd640 1 -- 172.21.15.84:0/3625144426 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5300af7b0 con 0x7fc5300a9b30 2024-03-29T10:37:35.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.857+0000 7fc52effd640 1 -- 172.21.15.84:0/3625144426 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 566327507 0 0) 0x7fc5300af7b0 con 0x7fc5300a9b30 2024-03-29T10:37:35.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.857+0000 7fc52effd640 1 -- 172.21.15.84:0/3625144426 >> v1:172.21.15.84:6790/0 conn(0x7fc5300a9750 legacy=0x7fc5300ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:35.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.857+0000 7fc52effd640 1 -- 172.21.15.84:0/3625144426 >> v1:172.21.15.86:6789/0 conn(0x7fc5300aad20 legacy=0x7fc530141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:35.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.857+0000 7fc52effd640 10 monclient: _finish_hunting 0 2024-03-29T10:37:35.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.857+0000 7fc52effd640 1 monclient: found mon.noname-a 2024-03-29T10:37:35.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.857+0000 7fc52effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:35.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.857+0000 7fc52effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T10:37:35.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.857+0000 7fc52effd640 1 -- 172.21.15.84:0/3625144426 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5300aba30 con 0x7fc5300a9b30 2024-03-29T10:37:35.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.857+0000 7fc52effd640 10 monclient: _finish_auth 0 2024-03-29T10:37:35.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.857+0000 7fc52effd640 10 monclient: _check_auth_tickets 2024-03-29T10:37:35.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.857+0000 7fc52effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:35.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.857+0000 7fc535419640 5 monclient: authenticate success, global_id 4326 2024-03-29T10:37:35.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.857+0000 7fc535419640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:37:35.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.857+0000 7fc52effd640 1 -- 172.21.15.84:0/3625144426 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fc528001930 con 0x7fc5300a9b30 2024-03-29T10:37:35.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.857+0000 7fc52effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:35.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.857+0000 7fc52effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:37:35.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.857+0000 7fc52effd640 10 monclient: dump: 2024-03-29T10:37:35.849 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:35.849 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:35.849 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:35.849 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:35.849 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:35.849 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:35.850 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:35.850 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:35.850 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:35.850 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:35.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.857+0000 7fc535419640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:37:35.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.857+0000 7fc52effd640 1 -- 172.21.15.84:0/3625144426 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc528001b50 con 0x7fc5300a9b30 2024-03-29T10:37:35.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.857+0000 7fc52effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:35.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.857+0000 7fc535419640 10 monclient: get_monmap_and_config success 2024-03-29T10:37:35.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.857+0000 7fc535419640 4 set_mon_vals no callback set 2024-03-29T10:37:35.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.857+0000 7fc52effd640 1 -- 172.21.15.84:0/3625144426 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fc528001ee0 con 0x7fc5300a9b30 2024-03-29T10:37:35.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.857+0000 7fc52effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:35.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.857+0000 7fc52effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:37:35.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.857+0000 7fc52effd640 10 monclient: dump: 2024-03-29T10:37:35.851 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:35.851 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:35.851 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:35.851 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:35.851 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:35.851 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:35.851 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:35.851 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:35.851 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:35.851 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:35.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.858+0000 7fc535419640 10 monclient: shutdown 2024-03-29T10:37:35.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.858+0000 7fc535419640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:35.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.858+0000 7fc535419640 1 -- 172.21.15.84:0/3625144426 >> v1:172.21.15.84:6789/0 conn(0x7fc5300a9b30 legacy=0x7fc530149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:35.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.858+0000 7fc535419640 1 -- 172.21.15.84:0/3625144426 shutdown_connections 2024-03-29T10:37:35.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.858+0000 7fc535419640 1 -- 172.21.15.84:0/3625144426 >> 172.21.15.84:0/3625144426 conn(0x7fc5300673e0 msgr2=0x7fc5300b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:35.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.858+0000 7fc535419640 1 -- 172.21.15.84:0/3625144426 shutdown_connections 2024-03-29T10:37:35.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.858+0000 7fc535419640 1 -- 172.21.15.84:0/3625144426 wait complete. 2024-03-29T10:37:35.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.859+0000 7fc535419640 10 monclient: build_initial_monmap 2024-03-29T10:37:35.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.859+0000 7fc535419640 10 monclient: monmap: 2024-03-29T10:37:35.852 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:37:35.852 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:37:35.852 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:37:35.852 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:37:35.852 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:35.852 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:35.852 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:37:35.852 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:37:35.852 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:37:35.853 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:35.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.860+0000 7fc535419640 1 librados: starting msgr at 2024-03-29T10:37:35.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.860+0000 7fc535419640 1 librados: starting objecter 2024-03-29T10:37:35.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.860+0000 7fc535419640 1 Processor -- start 2024-03-29T10:37:35.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.860+0000 7fc535419640 1 -- start start 2024-03-29T10:37:35.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.860+0000 7fc535419640 1 librados: setting wanted keys 2024-03-29T10:37:35.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.860+0000 7fc535419640 1 librados: calling monclient init 2024-03-29T10:37:35.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.860+0000 7fc535419640 10 monclient: init 2024-03-29T10:37:35.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.861+0000 7fc535419640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:35.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.861+0000 7fc535419640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:37:35.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.861+0000 7fc535419640 10 monclient(hunting): picked mon.noname-c con 0x7fc5300a9750 addr v1:172.21.15.84:6790/0 2024-03-29T10:37:35.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.862+0000 7fc535419640 10 monclient(hunting): picked mon.noname-b con 0x7fc5300a9b30 addr v1:172.21.15.86:6789/0 2024-03-29T10:37:35.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.862+0000 7fc535419640 10 monclient(hunting): picked mon.noname-a con 0x7fc5300aad20 addr v1:172.21.15.84:6789/0 2024-03-29T10:37:35.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.864+0000 7fc52f7fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc5300a9b30 0x7fc53006dc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:50421/0 (socket says 172.21.15.84:50421) 2024-03-29T10:37:35.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.864+0000 7fc52f7fe640 1 -- 172.21.15.84:0/3698013088 learned_addr learned my addr 172.21.15.84:0/3698013088 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:37:35.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.862+0000 7fc52ffff640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fc5300a9750 0x7fc5301d3270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:40556/0 (socket says 172.21.15.84:40556) 2024-03-29T10:37:35.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.864+0000 7fc52cff9640 10 client.?.objecter ms_handle_connect 0x7fc5300a9750 2024-03-29T10:37:35.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.864+0000 7fc52cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:35.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.864+0000 7fc535419640 1 -- 172.21.15.84:0/3698013088 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5301d2de0 con 0x7fc5300aad20 2024-03-29T10:37:35.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.864+0000 7fc52cff9640 10 client.?.objecter ms_handle_connect 0x7fc5300a9b30 2024-03-29T10:37:35.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.865+0000 7fc52cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:35.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.865+0000 7fc52cff9640 10 client.?.objecter ms_handle_connect 0x7fc5300aad20 2024-03-29T10:37:35.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.865+0000 7fc52cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:35.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.865+0000 7fc52cff9640 1 -- 172.21.15.84:0/3698013088 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121054700 0 0) 0x7fc5301d2de0 con 0x7fc5300aad20 2024-03-29T10:37:35.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.864+0000 7fc535419640 1 -- 172.21.15.84:0/3698013088 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc530071ae0 con 0x7fc5300a9b30 2024-03-29T10:37:35.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.868+0000 7fc535419640 1 -- 172.21.15.84:0/3698013088 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5301d2900 con 0x7fc5300a9750 2024-03-29T10:37:35.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.868+0000 7fc535419640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:35.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.868+0000 7fc535419640 10 monclient(hunting): authenticate will time out at 958.132076s 2024-03-29T10:37:35.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.869+0000 7fc52cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:37:35.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.869+0000 7fc52cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:35.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.869+0000 7fc52cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:35.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.869+0000 7fc52cff9640 10 monclient(hunting): my global_id is 4329 2024-03-29T10:37:35.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.869+0000 7fc52cff9640 1 -- 172.21.15.84:0/3698013088 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc51c004eb0 con 0x7fc5300aad20 2024-03-29T10:37:35.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.869+0000 7fc52cff9640 1 -- 172.21.15.84:0/3698013088 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2257955117 0 0) 0x7fc530071ae0 con 0x7fc5300a9b30 2024-03-29T10:37:35.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.869+0000 7fc52cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:37:35.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.869+0000 7fc52cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:35.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.869+0000 7fc52cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:35.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.869+0000 7fc52cff9640 10 monclient(hunting): my global_id is 4318 2024-03-29T10:37:35.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.869+0000 7fc52cff9640 1 -- 172.21.15.84:0/3698013088 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5301d2de0 con 0x7fc5300a9b30 2024-03-29T10:37:35.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 1 -- 172.21.15.84:0/3698013088 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3934780376 0 0) 0x7fc5301d2de0 con 0x7fc5300a9b30 2024-03-29T10:37:35.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 1 -- 172.21.15.84:0/3698013088 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc530071ae0 con 0x7fc5300a9b30 2024-03-29T10:37:35.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 1 -- 172.21.15.84:0/3698013088 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500407896 0 0) 0x7fc5301d2900 con 0x7fc5300a9750 2024-03-29T10:37:35.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:37:35.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:35.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:35.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 10 monclient(hunting): my global_id is 4352 2024-03-29T10:37:35.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 1 -- 172.21.15.84:0/3698013088 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5301d2de0 con 0x7fc5300a9750 2024-03-29T10:37:35.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 1 -- 172.21.15.84:0/3698013088 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fc520001950 con 0x7fc5300a9b30 2024-03-29T10:37:35.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:35.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:37:35.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 10 monclient(hunting): dump: 2024-03-29T10:37:35.857 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:35.857 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:35.857 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:35.858 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:35.858 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:35.858 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:35.858 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:35.858 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:35.858 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:35.858 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:35.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:37:35.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 1 -- 172.21.15.84:0/3698013088 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3874562331 0 0) 0x7fc5301d2de0 con 0x7fc5300a9750 2024-03-29T10:37:35.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 1 -- 172.21.15.84:0/3698013088 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5301d2900 con 0x7fc5300a9750 2024-03-29T10:37:35.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 1 -- 172.21.15.84:0/3698013088 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2003580316 0 0) 0x7fc530071ae0 con 0x7fc5300a9b30 2024-03-29T10:37:35.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 1 -- 172.21.15.84:0/3698013088 >> v1:172.21.15.84:6790/0 conn(0x7fc5300a9750 legacy=0x7fc5301d3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:35.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 1 -- 172.21.15.84:0/3698013088 >> v1:172.21.15.84:6789/0 conn(0x7fc5300aad20 legacy=0x7fc5301dd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:35.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:37:35.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 1 monclient: found mon.b 2024-03-29T10:37:35.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:35.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:37:35.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 1 -- 172.21.15.84:0/3698013088 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5301d2a90 con 0x7fc5300a9b30 2024-03-29T10:37:35.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 10 monclient: _finish_auth 0 2024-03-29T10:37:35.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:37:35.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:35.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc535419640 5 monclient: authenticate success, global_id 4318 2024-03-29T10:37:35.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc535419640 10 monclient: _renew_subs 2024-03-29T10:37:35.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc535419640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:37:35.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc535419640 1 -- 172.21.15.84:0/3698013088 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc53013b7c0 con 0x7fc5300a9b30 2024-03-29T10:37:35.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 1 -- 172.21.15.84:0/3698013088 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5200035f0 con 0x7fc5300a9b30 2024-03-29T10:37:35.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:35.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 1 -- 172.21.15.84:0/3698013088 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fc520003980 con 0x7fc5300a9b30 2024-03-29T10:37:35.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:35.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:37:35.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc52cff9640 10 monclient: dump: 2024-03-29T10:37:35.861 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:35.861 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:35.861 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:35.861 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:35.862 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:35.862 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:35.862 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:35.862 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:35.862 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:35.862 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:35.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc535419640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:37:35.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc535419640 10 monclient: _renew_subs 2024-03-29T10:37:35.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc535419640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:37:35.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc535419640 1 -- 172.21.15.84:0/3698013088 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc53013f2b0 con 0x7fc5300a9b30 2024-03-29T10:37:35.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.870+0000 7fc535419640 1 librados: init done 2024-03-29T10:37:35.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.871+0000 7fc52dffb640 4 set_mon_vals no callback set 2024-03-29T10:37:35.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.872+0000 7fc52cff9640 1 -- 172.21.15.84:0/3698013088 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7fc520003ae0 con 0x7fc5300a9b30 2024-03-29T10:37:35.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.872+0000 7fc535419640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:35.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.873+0000 7fc52cff9640 1 -- 172.21.15.84:0/3698013088 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7fc520084910 con 0x7fc5300a9b30 2024-03-29T10:37:35.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.873+0000 7fc52cff9640 10 client.4318.objecter ms_dispatch 0x7fc530000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:37:35.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.873+0000 7fc52cff9640 3 client.4318.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:37:35.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.873+0000 7fc52cff9640 3 client.4318.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:37:35.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.873+0000 7fc52cff9640 20 client.4318.objecter dump_active .. 0 homeless 2024-03-29T10:37:35.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.874+0000 7fc535419640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:35.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.874+0000 7fc535419640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:37:35.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.874+0000 7fc535419640 1 -- 172.21.15.84:0/3698013088 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc510005080 con 0x7fc5300a9b30 2024-03-29T10:37:35.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.874+0000 7fc52ffff640 10 monclient: get_auth_request con 0x7fc51c06df40 auth_method 0 2024-03-29T10:37:35.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.874+0000 7fc52cff9640 10 client.4318.objecter ms_handle_connect 0x7fc51c06df40 2024-03-29T10:37:35.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.884+0000 7fc52cff9640 1 -- 172.21.15.84:0/3698013088 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc520050270 con 0x7fc5300a9b30 2024-03-29T10:37:35.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.884+0000 7fc52cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:35.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:35.884+0000 7fc52cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:37:36.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:36.207+0000 7fc535419640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:36.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:36.207+0000 7fc535419640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:36.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:36.207+0000 7fc535419640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:37:36.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:36.207+0000 7fc535419640 1 -- 172.21.15.84:0/3698013088 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc510005330 con 0x7fc5300a9b30 2024-03-29T10:37:36.194 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:36.209+0000 7fc52cff9640 1 -- 172.21.15.84:0/3698013088 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7fc5200521a0 con 0x7fc5300a9b30 2024-03-29T10:37:36.194 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:36.209+0000 7fc52cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:36.194 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:36.209+0000 7fc52cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:37:36.194 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:37:36.194 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:37:36.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:36.214+0000 7fc535419640 10 librados: watch_flush enter 2024-03-29T10:37:36.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:36.215+0000 7fc535419640 10 librados: watch_flush exit 2024-03-29T10:37:36.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:36.215+0000 7fc535419640 20 client.4318.objecter shutdown clearing up homeless session... 2024-03-29T10:37:36.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:36.215+0000 7fc535419640 10 client.4318.objecter successfully canceled tick 2024-03-29T10:37:36.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:36.215+0000 7fc535419640 1 -- 172.21.15.84:0/3698013088 >> v1:172.21.15.86:6816/3337225337 conn(0x7fc51c06df40 legacy=0x7fc51c070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:36.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:36.215+0000 7fc535419640 10 monclient: shutdown 2024-03-29T10:37:36.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:36.215+0000 7fc535419640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:36.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:36.215+0000 7fc535419640 1 -- 172.21.15.84:0/3698013088 >> v1:172.21.15.86:6789/0 conn(0x7fc5300a9b30 legacy=0x7fc53006dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:36.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:36.215+0000 7fc535419640 1 -- 172.21.15.84:0/3698013088 shutdown_connections 2024-03-29T10:37:36.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:36.215+0000 7fc535419640 1 -- 172.21.15.84:0/3698013088 >> 172.21.15.84:0/3698013088 conn(0x7fc5300673e0 msgr2=0x7fc5300b1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:36.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:36.215+0000 7fc535419640 1 -- 172.21.15.84:0/3698013088 shutdown_connections 2024-03-29T10:37:36.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:36.216+0000 7fc535419640 1 -- 172.21.15.84:0/3698013088 wait complete. 2024-03-29T10:37:36.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:36.216+0000 7fc535419640 1 librados: shutdown 2024-03-29T10:37:36.219 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:37:43.026 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:37:43.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.187+0000 7f4ba50c0640 10 monclient: get_monmap_and_config 2024-03-29T10:37:43.172 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.187+0000 7f4ba50c0640 10 monclient: build_initial_monmap 2024-03-29T10:37:43.173 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.187+0000 7f4ba50c0640 10 monclient: monmap: 2024-03-29T10:37:43.173 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:37:43.173 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:43.173 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:37:43.189093+0000 2024-03-29T10:37:43.173 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:37:43.189093+0000 2024-03-29T10:37:43.173 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:43.173 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:43.173 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:37:43.173 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:37:43.173 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:37:43.173 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:43.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.188+0000 7f4ba50c0640 1 Processor -- start 2024-03-29T10:37:43.174 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.188+0000 7f4ba50c0640 1 -- start start 2024-03-29T10:37:43.176 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.188+0000 7f4ba50c0640 10 monclient: init 2024-03-29T10:37:43.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.188+0000 7f4ba50c0640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:43.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.188+0000 7f4ba50c0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:37:43.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.188+0000 7f4ba50c0640 10 monclient(hunting): picked mon.noname-a con 0x7f4ba01457f0 addr v1:172.21.15.84:6789/0 2024-03-29T10:37:43.178 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.189+0000 7f4ba50c0640 10 monclient(hunting): picked mon.noname-b con 0x7f4ba013f650 addr v1:172.21.15.84:6790/0 2024-03-29T10:37:43.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.191+0000 7f4ba50c0640 10 monclient(hunting): picked mon.noname-c con 0x7f4ba00a9d70 addr v1:172.21.15.86:6789/0 2024-03-29T10:37:43.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.191+0000 7f4ba50c0640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba00aaef0 con 0x7f4ba01457f0 2024-03-29T10:37:43.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.191+0000 7f4ba50c0640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba0145bd0 con 0x7f4ba00a9d70 2024-03-29T10:37:43.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.192+0000 7f4b9e59c640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f4ba013f650 0x7f4ba00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:51454/0 (socket says 172.21.15.84:51454) 2024-03-29T10:37:43.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.192+0000 7f4b9ed9d640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f4ba01457f0 0x7f4ba013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:45917/0 (socket says 172.21.15.84:45917) 2024-03-29T10:37:43.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.192+0000 7f4b9e59c640 1 -- 172.21.15.84:0/1060615037 learned_addr learned my addr 172.21.15.84:0/1060615037 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:37:43.179 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.192+0000 7f4ba50c0640 1 -- 172.21.15.84:0/1060615037 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba0145eb0 con 0x7f4ba013f650 2024-03-29T10:37:43.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.192+0000 7f4ba50c0640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:43.180 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.192+0000 7f4ba50c0640 10 monclient(hunting): authenticate will time out at 965.456685s 2024-03-29T10:37:43.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.193+0000 7f4b9dd9b640 1 -- 172.21.15.84:0/1060615037 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2118096262 0 0) 0x7f4ba00aaef0 con 0x7f4ba01457f0 2024-03-29T10:37:43.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.193+0000 7f4b9dd9b640 10 monclient(hunting): _negotiate 2024-03-29T10:37:43.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.193+0000 7f4b9dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:43.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.193+0000 7f4b9dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:43.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.193+0000 7f4b9dd9b640 10 monclient(hunting): my global_id is 4332 2024-03-29T10:37:43.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.193+0000 7f4b9dd9b640 1 -- 172.21.15.84:0/1060615037 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b88003520 con 0x7f4ba01457f0 2024-03-29T10:37:43.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.193+0000 7f4b9dd9b640 1 -- 172.21.15.84:0/1060615037 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228886534 0 0) 0x7f4ba0145bd0 con 0x7f4ba00a9d70 2024-03-29T10:37:43.184 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.194+0000 7f4b9dd9b640 10 monclient(hunting): _negotiate 2024-03-29T10:37:43.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.194+0000 7f4b9dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:43.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.194+0000 7f4b9dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:43.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.194+0000 7f4b9dd9b640 10 monclient(hunting): my global_id is 4321 2024-03-29T10:37:43.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.194+0000 7f4b9dd9b640 1 -- 172.21.15.84:0/1060615037 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ba00aaef0 con 0x7f4ba00a9d70 2024-03-29T10:37:43.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.194+0000 7f4b9dd9b640 1 -- 172.21.15.84:0/1060615037 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1943943251 0 0) 0x7f4ba00aaef0 con 0x7f4ba00a9d70 2024-03-29T10:37:43.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.194+0000 7f4b9dd9b640 1 -- 172.21.15.84:0/1060615037 >> v1:172.21.15.84:6790/0 conn(0x7f4ba013f650 legacy=0x7f4ba00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:43.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.194+0000 7f4b9dd9b640 1 -- 172.21.15.84:0/1060615037 >> v1:172.21.15.84:6789/0 conn(0x7f4ba01457f0 legacy=0x7f4ba013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:43.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.194+0000 7f4b9dd9b640 10 monclient: _finish_hunting 0 2024-03-29T10:37:43.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.194+0000 7f4b9dd9b640 1 monclient: found mon.noname-c 2024-03-29T10:37:43.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.194+0000 7f4b9dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:43.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.194+0000 7f4b9dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-29T10:37:43.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.194+0000 7f4b9dd9b640 1 -- 172.21.15.84:0/1060615037 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba0146340 con 0x7f4ba00a9d70 2024-03-29T10:37:43.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.194+0000 7f4b9dd9b640 10 monclient: _finish_auth 0 2024-03-29T10:37:43.185 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.194+0000 7f4b9dd9b640 10 monclient: _check_auth_tickets 2024-03-29T10:37:43.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.194+0000 7f4b9dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:43.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.194+0000 7f4b9dd9b640 1 -- 172.21.15.84:0/1060615037 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4b98001920 con 0x7f4ba00a9d70 2024-03-29T10:37:43.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.194+0000 7f4b9dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:43.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.194+0000 7f4b9dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:37:43.186 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.194+0000 7f4b9dd9b640 10 monclient: dump: 2024-03-29T10:37:43.186 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:43.186 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:43.186 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:43.186 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:43.186 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:43.186 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:43.186 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:43.187 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:43.187 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:43.187 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:43.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.194+0000 7f4ba50c0640 5 monclient: authenticate success, global_id 4321 2024-03-29T10:37:43.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.194+0000 7f4ba50c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:37:43.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.198+0000 7f4b9dd9b640 1 -- 172.21.15.84:0/1060615037 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b98001a80 con 0x7f4ba00a9d70 2024-03-29T10:37:43.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.198+0000 7f4b9dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:43.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.198+0000 7f4b9dd9b640 1 -- 172.21.15.84:0/1060615037 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4b98001e90 con 0x7f4ba00a9d70 2024-03-29T10:37:43.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.198+0000 7f4b9dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:43.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.198+0000 7f4b9dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:37:43.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.198+0000 7f4b9dd9b640 10 monclient: dump: 2024-03-29T10:37:43.187 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:43.187 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:43.188 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:43.188 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:43.188 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:43.188 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:43.188 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:43.188 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:43.188 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:43.188 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:43.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.198+0000 7f4ba50c0640 10 monclient: get_monmap_and_config success 2024-03-29T10:37:43.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.198+0000 7f4ba50c0640 4 set_mon_vals no callback set 2024-03-29T10:37:43.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.199+0000 7f4ba50c0640 10 monclient: shutdown 2024-03-29T10:37:43.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.199+0000 7f4ba50c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:43.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.199+0000 7f4ba50c0640 1 -- 172.21.15.84:0/1060615037 >> v1:172.21.15.86:6789/0 conn(0x7f4ba00a9d70 legacy=0x7f4ba014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:43.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.200+0000 7f4ba50c0640 1 -- 172.21.15.84:0/1060615037 shutdown_connections 2024-03-29T10:37:43.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.200+0000 7f4ba50c0640 1 -- 172.21.15.84:0/1060615037 >> 172.21.15.84:0/1060615037 conn(0x7f4ba0066a00 msgr2=0x7f4ba0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:43.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.201+0000 7f4ba50c0640 1 -- 172.21.15.84:0/1060615037 shutdown_connections 2024-03-29T10:37:43.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.201+0000 7f4ba50c0640 1 -- 172.21.15.84:0/1060615037 wait complete. 2024-03-29T10:37:43.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.203+0000 7f4ba50c0640 10 monclient: build_initial_monmap 2024-03-29T10:37:43.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.203+0000 7f4ba50c0640 10 monclient: monmap: 2024-03-29T10:37:43.190 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:37:43.190 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:37:43.190 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:37:43.190 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:37:43.190 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:43.190 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:43.190 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:37:43.190 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:37:43.190 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:37:43.191 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:43.191 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.203+0000 7f4ba50c0640 1 librados: starting msgr at 2024-03-29T10:37:43.191 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.203+0000 7f4ba50c0640 1 librados: starting objecter 2024-03-29T10:37:43.191 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.203+0000 7f4ba50c0640 1 Processor -- start 2024-03-29T10:37:43.191 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.203+0000 7f4ba50c0640 1 -- start start 2024-03-29T10:37:43.191 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.203+0000 7f4ba50c0640 1 librados: setting wanted keys 2024-03-29T10:37:43.191 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.203+0000 7f4ba50c0640 1 librados: calling monclient init 2024-03-29T10:37:43.191 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.203+0000 7f4ba50c0640 10 monclient: init 2024-03-29T10:37:43.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.204+0000 7f4ba50c0640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:43.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.204+0000 7f4ba50c0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:37:43.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.204+0000 7f4ba50c0640 10 monclient(hunting): picked mon.noname-b con 0x7f4ba00a9d70 addr v1:172.21.15.86:6789/0 2024-03-29T10:37:43.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.204+0000 7f4ba50c0640 10 monclient(hunting): picked mon.noname-c con 0x7f4ba013f650 addr v1:172.21.15.84:6790/0 2024-03-29T10:37:43.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.204+0000 7f4ba50c0640 10 monclient(hunting): picked mon.noname-a con 0x7f4ba01457f0 addr v1:172.21.15.84:6789/0 2024-03-29T10:37:43.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.204+0000 7f4ba50c0640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba00ada10 con 0x7f4ba01457f0 2024-03-29T10:37:43.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.204+0000 7f4ba50c0640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba00b1800 con 0x7f4ba00a9d70 2024-03-29T10:37:43.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.204+0000 7f4ba50c0640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba00abd10 con 0x7f4ba013f650 2024-03-29T10:37:43.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.204+0000 7f4ba50c0640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:43.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.204+0000 7f4ba50c0640 10 monclient(hunting): authenticate will time out at 965.469055s 2024-03-29T10:37:43.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.205+0000 7f4b9e59c640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f4ba013f650 0x7f4ba00b09e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:53890/0 (socket says 172.21.15.84:53890) 2024-03-29T10:37:43.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.205+0000 7f4b9e59c640 1 -- 172.21.15.84:0/383986361 learned_addr learned my addr 172.21.15.84:0/383986361 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:37:43.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.207+0000 7f4b877fe640 10 client.?.objecter ms_handle_connect 0x7f4ba013f650 2024-03-29T10:37:43.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.207+0000 7f4b877fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:43.194 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.207+0000 7f4b877fe640 1 -- 172.21.15.84:0/383986361 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3672282438 0 0) 0x7f4ba00abd10 con 0x7f4ba013f650 2024-03-29T10:37:43.194 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.207+0000 7f4b877fe640 10 monclient(hunting): _negotiate 2024-03-29T10:37:43.194 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.207+0000 7f4b877fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:43.194 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.207+0000 7f4b877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:43.194 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.207+0000 7f4b877fe640 10 monclient(hunting): my global_id is 4355 2024-03-29T10:37:43.194 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.207+0000 7f4b877fe640 1 -- 172.21.15.84:0/383986361 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b8c004eb0 con 0x7f4ba013f650 2024-03-29T10:37:43.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.208+0000 7f4b877fe640 1 -- 172.21.15.84:0/383986361 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2325351018 0 0) 0x7f4b8c004eb0 con 0x7f4ba013f650 2024-03-29T10:37:43.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.208+0000 7f4b877fe640 1 -- 172.21.15.84:0/383986361 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ba00abd10 con 0x7f4ba013f650 2024-03-29T10:37:43.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.208+0000 7f4b877fe640 10 client.?.objecter ms_handle_connect 0x7f4ba00a9d70 2024-03-29T10:37:43.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.208+0000 7f4b877fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:43.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.208+0000 7f4b877fe640 1 -- 172.21.15.84:0/383986361 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4b940018b0 con 0x7f4ba013f650 2024-03-29T10:37:43.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.208+0000 7f4b877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:43.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.208+0000 7f4b877fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:37:43.197 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.208+0000 7f4b877fe640 10 monclient(hunting): dump: 2024-03-29T10:37:43.197 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:43.197 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:43.197 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:43.197 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:43.197 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:43.197 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:43.197 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:43.197 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:43.197 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:43.198 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:43.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.208+0000 7f4b877fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:37:43.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.208+0000 7f4b877fe640 10 client.?.objecter ms_handle_connect 0x7f4ba01457f0 2024-03-29T10:37:43.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.208+0000 7f4b877fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:43.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b877fe640 1 -- 172.21.15.84:0/383986361 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 166186749 0 0) 0x7f4ba00ada10 con 0x7f4ba01457f0 2024-03-29T10:37:43.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b877fe640 10 monclient(hunting): _negotiate 2024-03-29T10:37:43.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b877fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:43.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:43.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b877fe640 10 monclient(hunting): my global_id is 4335 2024-03-29T10:37:43.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b877fe640 1 -- 172.21.15.84:0/383986361 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b8c004eb0 con 0x7f4ba01457f0 2024-03-29T10:37:43.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b877fe640 1 -- 172.21.15.84:0/383986361 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129768099 0 0) 0x7f4ba00b1800 con 0x7f4ba00a9d70 2024-03-29T10:37:43.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b877fe640 10 monclient(hunting): _negotiate 2024-03-29T10:37:43.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b877fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:43.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:43.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b877fe640 10 monclient(hunting): my global_id is 4324 2024-03-29T10:37:43.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b877fe640 1 -- 172.21.15.84:0/383986361 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ba00ada10 con 0x7f4ba00a9d70 2024-03-29T10:37:43.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b877fe640 1 -- 172.21.15.84:0/383986361 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3071230000 0 0) 0x7f4ba00abd10 con 0x7f4ba013f650 2024-03-29T10:37:43.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b877fe640 1 -- 172.21.15.84:0/383986361 >> v1:172.21.15.86:6789/0 conn(0x7f4ba00a9d70 legacy=0x7f4ba00ab600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:43.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b877fe640 1 -- 172.21.15.84:0/383986361 >> v1:172.21.15.84:6789/0 conn(0x7f4ba01457f0 legacy=0x7f4ba00b10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:43.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b877fe640 10 monclient: _finish_hunting 0 2024-03-29T10:37:43.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b877fe640 1 monclient: found mon.c 2024-03-29T10:37:43.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:43.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b877fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:37:43.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b877fe640 1 -- 172.21.15.84:0/383986361 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba00ad5a0 con 0x7f4ba013f650 2024-03-29T10:37:43.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b877fe640 10 monclient: _finish_auth 0 2024-03-29T10:37:43.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b877fe640 10 monclient: _check_auth_tickets 2024-03-29T10:37:43.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:43.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b877fe640 1 -- 172.21.15.84:0/383986361 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b94001a80 con 0x7f4ba013f650 2024-03-29T10:37:43.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:43.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b877fe640 1 -- 172.21.15.84:0/383986361 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4b94004a90 con 0x7f4ba013f650 2024-03-29T10:37:43.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:43.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:37:43.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b877fe640 10 monclient: dump: 2024-03-29T10:37:43.200 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:43.200 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:43.200 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:43.201 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:43.201 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:43.201 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:43.201 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:43.201 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:43.201 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:43.201 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:43.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.209+0000 7f4b9d59a640 4 set_mon_vals no callback set 2024-03-29T10:37:43.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.210+0000 7f4ba50c0640 5 monclient: authenticate success, global_id 4355 2024-03-29T10:37:43.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.210+0000 7f4ba50c0640 10 monclient: _renew_subs 2024-03-29T10:37:43.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.210+0000 7f4ba50c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:37:43.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.210+0000 7f4ba50c0640 1 -- 172.21.15.84:0/383986361 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba00adea0 con 0x7f4ba013f650 2024-03-29T10:37:43.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.210+0000 7f4ba50c0640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:37:43.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.210+0000 7f4ba50c0640 10 monclient: _renew_subs 2024-03-29T10:37:43.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.210+0000 7f4ba50c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:37:43.202 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.210+0000 7f4ba50c0640 1 -- 172.21.15.84:0/383986361 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba00ad2c0 con 0x7f4ba013f650 2024-03-29T10:37:43.202 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.210+0000 7f4ba50c0640 1 librados: init done 2024-03-29T10:37:43.203 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.212+0000 7f4b877fe640 1 -- 172.21.15.84:0/383986361 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f4b94004bf0 con 0x7f4ba013f650 2024-03-29T10:37:43.203 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.212+0000 7f4b857fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:43.203 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.212+0000 7f4b857fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:43.203 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.212+0000 7f4b857fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:37:43.203 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.212+0000 7f4b857fa640 1 -- 172.21.15.84:0/383986361 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b78005080 con 0x7f4ba013f650 2024-03-29T10:37:43.203 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.213+0000 7f4b877fe640 1 -- 172.21.15.84:0/383986361 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f4b94084b00 con 0x7f4ba013f650 2024-03-29T10:37:43.203 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.213+0000 7f4b877fe640 10 client.4355.objecter ms_dispatch 0x7f4ba0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:37:43.203 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.213+0000 7f4b877fe640 3 client.4355.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:37:43.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.213+0000 7f4b877fe640 3 client.4355.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:37:43.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.213+0000 7f4b9ed9d640 10 monclient: get_auth_request con 0x7f4b8c06dd60 auth_method 0 2024-03-29T10:37:43.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.213+0000 7f4b877fe640 20 client.4355.objecter dump_active .. 0 homeless 2024-03-29T10:37:43.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.216+0000 7f4b877fe640 10 client.4355.objecter ms_handle_connect 0x7f4b8c06dd60 2024-03-29T10:37:43.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.222+0000 7f4b877fe640 1 -- 172.21.15.84:0/383986361 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4b940503e0 con 0x7f4ba013f650 2024-03-29T10:37:43.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.222+0000 7f4b877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:43.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.222+0000 7f4b877fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:37:43.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.549+0000 7f4b857fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:43.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.549+0000 7f4b857fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:43.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.549+0000 7f4b857fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:37:43.535 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.549+0000 7f4b857fa640 1 -- 172.21.15.84:0/383986361 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b78005330 con 0x7f4ba013f650 2024-03-29T10:37:43.535 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.550+0000 7f4b877fe640 1 -- 172.21.15.84:0/383986361 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f4b94052310 con 0x7f4ba013f650 2024-03-29T10:37:43.535 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.550+0000 7f4b877fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:43.535 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.550+0000 7f4b877fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:37:43.537 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:37:43.537 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:37:43.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.555+0000 7f4b857fa640 10 librados: watch_flush enter 2024-03-29T10:37:43.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.555+0000 7f4b857fa640 10 librados: watch_flush exit 2024-03-29T10:37:43.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.555+0000 7f4b857fa640 20 client.4355.objecter shutdown clearing up homeless session... 2024-03-29T10:37:43.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.555+0000 7f4b857fa640 10 client.4355.objecter successfully canceled tick 2024-03-29T10:37:43.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.555+0000 7f4b857fa640 1 -- 172.21.15.84:0/383986361 >> v1:172.21.15.86:6816/3337225337 conn(0x7f4b8c06dd60 legacy=0x7f4b8c070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:43.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.555+0000 7f4b857fa640 10 monclient: shutdown 2024-03-29T10:37:43.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.555+0000 7f4b857fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:43.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.555+0000 7f4b857fa640 1 -- 172.21.15.84:0/383986361 >> v1:172.21.15.84:6790/0 conn(0x7f4ba013f650 legacy=0x7f4ba00b09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:43.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.555+0000 7f4b857fa640 1 -- 172.21.15.84:0/383986361 shutdown_connections 2024-03-29T10:37:43.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.555+0000 7f4b857fa640 1 -- 172.21.15.84:0/383986361 >> 172.21.15.84:0/383986361 conn(0x7f4ba0066a00 msgr2=0x7f4ba0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:43.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.556+0000 7f4b857fa640 1 -- 172.21.15.84:0/383986361 shutdown_connections 2024-03-29T10:37:43.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.556+0000 7f4b857fa640 1 -- 172.21.15.84:0/383986361 wait complete. 2024-03-29T10:37:43.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:43.556+0000 7f4b857fa640 1 librados: shutdown 2024-03-29T10:37:43.558 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:37:50.364 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:37:50.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.525+0000 7f4e562fb640 10 monclient: get_monmap_and_config 2024-03-29T10:37:50.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.525+0000 7f4e562fb640 10 monclient: build_initial_monmap 2024-03-29T10:37:50.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.525+0000 7f4e562fb640 10 monclient: monmap: 2024-03-29T10:37:50.510 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:37:50.510 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:50.510 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:37:50.526182+0000 2024-03-29T10:37:50.510 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:37:50.526182+0000 2024-03-29T10:37:50.510 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:50.510 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:50.510 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:37:50.510 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:37:50.510 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:37:50.510 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:50.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.527+0000 7f4e562fb640 1 Processor -- start 2024-03-29T10:37:50.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.527+0000 7f4e562fb640 1 -- start start 2024-03-29T10:37:50.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.527+0000 7f4e562fb640 10 monclient: init 2024-03-29T10:37:50.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.528+0000 7f4e562fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:50.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.528+0000 7f4e562fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:37:50.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.528+0000 7f4e562fb640 10 monclient(hunting): picked mon.noname-c con 0x7f4e501457f0 addr v1:172.21.15.86:6789/0 2024-03-29T10:37:50.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.528+0000 7f4e562fb640 10 monclient(hunting): picked mon.noname-a con 0x7f4e5013f650 addr v1:172.21.15.84:6789/0 2024-03-29T10:37:50.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.528+0000 7f4e562fb640 10 monclient(hunting): picked mon.noname-b con 0x7f4e500a9d70 addr v1:172.21.15.84:6790/0 2024-03-29T10:37:50.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.528+0000 7f4e562fb640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e500aaef0 con 0x7f4e5013f650 2024-03-29T10:37:50.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.528+0000 7f4e562fb640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e50145bd0 con 0x7f4e501457f0 2024-03-29T10:37:50.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.528+0000 7f4e562fb640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e50145eb0 con 0x7f4e500a9d70 2024-03-29T10:37:50.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.528+0000 7f4e562fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:50.514 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.528+0000 7f4e562fb640 10 monclient(hunting): authenticate will time out at 972.791789s 2024-03-29T10:37:50.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.535+0000 7f4e4f7fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f4e5013f650 0x7f4e500b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:37173/0 (socket says 172.21.15.84:37173) 2024-03-29T10:37:50.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.535+0000 7f4e4ffff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f4e501457f0 0x7f4e5013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:40307/0 (socket says 172.21.15.84:40307) 2024-03-29T10:37:50.519 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.535+0000 7f4e4ffff640 1 -- 172.21.15.84:0/3563695602 learned_addr learned my addr 172.21.15.84:0/3563695602 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:37:50.520 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.535+0000 7f4e4effd640 1 -- 172.21.15.84:0/3563695602 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 945990580 0 0) 0x7f4e50145bd0 con 0x7f4e501457f0 2024-03-29T10:37:50.520 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.535+0000 7f4e4effd640 10 monclient(hunting): _negotiate 2024-03-29T10:37:50.520 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.535+0000 7f4e4effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:50.520 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.535+0000 7f4e4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:50.520 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.535+0000 7f4e4effd640 10 monclient(hunting): my global_id is 4327 2024-03-29T10:37:50.520 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.535+0000 7f4e4effd640 1 -- 172.21.15.84:0/3563695602 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e38003520 con 0x7f4e501457f0 2024-03-29T10:37:50.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.538+0000 7f4e4effd640 1 -- 172.21.15.84:0/3563695602 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4135566201 0 0) 0x7f4e38003520 con 0x7f4e501457f0 2024-03-29T10:37:50.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.538+0000 7f4e4effd640 1 -- 172.21.15.84:0/3563695602 >> v1:172.21.15.84:6790/0 conn(0x7f4e500a9d70 legacy=0x7f4e5014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:50.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.538+0000 7f4e4effd640 1 -- 172.21.15.84:0/3563695602 >> v1:172.21.15.84:6789/0 conn(0x7f4e5013f650 legacy=0x7f4e500b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:50.523 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.539+0000 7f4e4effd640 10 monclient: _finish_hunting 0 2024-03-29T10:37:50.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.539+0000 7f4e4effd640 1 monclient: found mon.noname-c 2024-03-29T10:37:50.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.539+0000 7f4e4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:50.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.539+0000 7f4e4effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-29T10:37:50.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.539+0000 7f4e4effd640 1 -- 172.21.15.84:0/3563695602 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e50146340 con 0x7f4e501457f0 2024-03-29T10:37:50.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.539+0000 7f4e4effd640 10 monclient: _finish_auth 0 2024-03-29T10:37:50.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.539+0000 7f4e4effd640 10 monclient: _check_auth_tickets 2024-03-29T10:37:50.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.539+0000 7f4e4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:50.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.539+0000 7f4e4effd640 1 -- 172.21.15.84:0/3563695602 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4e40001690 con 0x7f4e501457f0 2024-03-29T10:37:50.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.539+0000 7f4e4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:50.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.539+0000 7f4e4effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:37:50.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.539+0000 7f4e4effd640 10 monclient: dump: 2024-03-29T10:37:50.524 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:50.524 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:50.524 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:50.524 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:50.525 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:50.525 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:50.525 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:50.525 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:50.525 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:50.525 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:50.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.539+0000 7f4e4effd640 1 -- 172.21.15.84:0/3563695602 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e400017f0 con 0x7f4e501457f0 2024-03-29T10:37:50.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.539+0000 7f4e4effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:50.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.539+0000 7f4e4effd640 1 -- 172.21.15.84:0/3563695602 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4e40001c00 con 0x7f4e501457f0 2024-03-29T10:37:50.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.539+0000 7f4e4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:50.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.539+0000 7f4e4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:37:50.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.539+0000 7f4e4effd640 10 monclient: dump: 2024-03-29T10:37:50.525 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:50.525 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:50.526 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:50.526 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:50.526 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:50.526 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:50.526 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:50.526 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:50.526 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:50.526 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:50.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.539+0000 7f4e562fb640 5 monclient: authenticate success, global_id 4327 2024-03-29T10:37:50.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.539+0000 7f4e562fb640 10 monclient: get_monmap_and_config success 2024-03-29T10:37:50.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.539+0000 7f4e562fb640 4 set_mon_vals no callback set 2024-03-29T10:37:50.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.540+0000 7f4e562fb640 10 monclient: shutdown 2024-03-29T10:37:50.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.540+0000 7f4e562fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:50.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.540+0000 7f4e562fb640 1 -- 172.21.15.84:0/3563695602 >> v1:172.21.15.86:6789/0 conn(0x7f4e501457f0 legacy=0x7f4e5013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:50.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.541+0000 7f4e562fb640 1 -- 172.21.15.84:0/3563695602 shutdown_connections 2024-03-29T10:37:50.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.541+0000 7f4e562fb640 1 -- 172.21.15.84:0/3563695602 >> 172.21.15.84:0/3563695602 conn(0x7f4e50066a00 msgr2=0x7f4e50066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:50.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.541+0000 7f4e562fb640 1 -- 172.21.15.84:0/3563695602 shutdown_connections 2024-03-29T10:37:50.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.543+0000 7f4e562fb640 1 -- 172.21.15.84:0/3563695602 wait complete. 2024-03-29T10:37:50.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.544+0000 7f4e562fb640 10 monclient: build_initial_monmap 2024-03-29T10:37:50.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.544+0000 7f4e562fb640 10 monclient: monmap: 2024-03-29T10:37:50.529 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:37:50.529 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:37:50.529 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:37:50.529 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:37:50.529 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:50.529 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:50.529 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:37:50.529 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:37:50.529 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:37:50.529 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:50.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.545+0000 7f4e562fb640 1 librados: starting msgr at 2024-03-29T10:37:50.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.545+0000 7f4e562fb640 1 librados: starting objecter 2024-03-29T10:37:50.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.545+0000 7f4e562fb640 1 Processor -- start 2024-03-29T10:37:50.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.545+0000 7f4e562fb640 1 -- start start 2024-03-29T10:37:50.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.546+0000 7f4e562fb640 1 librados: setting wanted keys 2024-03-29T10:37:50.531 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.546+0000 7f4e562fb640 1 librados: calling monclient init 2024-03-29T10:37:50.531 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.546+0000 7f4e562fb640 10 monclient: init 2024-03-29T10:37:50.531 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.547+0000 7f4e562fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:50.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.547+0000 7f4e562fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:37:50.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.547+0000 7f4e562fb640 10 monclient(hunting): picked mon.noname-b con 0x7f4e500a9d70 addr v1:172.21.15.86:6789/0 2024-03-29T10:37:50.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.547+0000 7f4e562fb640 10 monclient(hunting): picked mon.noname-a con 0x7f4e5013f650 addr v1:172.21.15.84:6789/0 2024-03-29T10:37:50.536 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.549+0000 7f4e562fb640 10 monclient(hunting): picked mon.noname-c con 0x7f4e501403c0 addr v1:172.21.15.84:6790/0 2024-03-29T10:37:50.536 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.550+0000 7f4e562fb640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e50141a80 con 0x7f4e5013f650 2024-03-29T10:37:50.536 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.550+0000 7f4e562fb640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e50142e00 con 0x7f4e500a9d70 2024-03-29T10:37:50.536 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.550+0000 7f4e54898640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f4e501403c0 0x7f4e501dac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:34095/0 (socket says 172.21.15.84:34095) 2024-03-29T10:37:50.536 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.550+0000 7f4e54898640 1 -- 172.21.15.84:0/992632367 learned_addr learned my addr 172.21.15.84:0/992632367 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:37:50.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.550+0000 7f4e4cff9640 10 client.?.objecter ms_handle_connect 0x7f4e501403c0 2024-03-29T10:37:50.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.550+0000 7f4e4cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:50.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.550+0000 7f4e562fb640 1 -- 172.21.15.84:0/992632367 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e501407e0 con 0x7f4e501403c0 2024-03-29T10:37:50.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.550+0000 7f4e562fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:50.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.550+0000 7f4e4cff9640 1 -- 172.21.15.84:0/992632367 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055620398 0 0) 0x7f4e501407e0 con 0x7f4e501403c0 2024-03-29T10:37:50.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.550+0000 7f4e562fb640 10 monclient(hunting): authenticate will time out at 972.814056s 2024-03-29T10:37:50.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.551+0000 7f4e4cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:37:50.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.551+0000 7f4e4cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:50.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.551+0000 7f4e4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:50.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.551+0000 7f4e4cff9640 10 monclient(hunting): my global_id is 4361 2024-03-29T10:37:50.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.551+0000 7f4e4cff9640 1 -- 172.21.15.84:0/992632367 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e3c004eb0 con 0x7f4e501403c0 2024-03-29T10:37:50.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.551+0000 7f4e4cff9640 1 -- 172.21.15.84:0/992632367 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3091467478 0 0) 0x7f4e3c004eb0 con 0x7f4e501403c0 2024-03-29T10:37:50.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.551+0000 7f4e4cff9640 1 -- 172.21.15.84:0/992632367 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e501407e0 con 0x7f4e501403c0 2024-03-29T10:37:50.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.551+0000 7f4e4cff9640 1 -- 172.21.15.84:0/992632367 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4e48001d10 con 0x7f4e501403c0 2024-03-29T10:37:50.538 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.551+0000 7f4e4cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:50.538 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.551+0000 7f4e4cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:37:50.538 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.552+0000 7f4e4cff9640 10 monclient(hunting): dump: 2024-03-29T10:37:50.538 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:50.538 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:50.538 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:50.538 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:50.538 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:50.538 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:50.538 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:50.538 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:50.538 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:50.538 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:50.538 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.552+0000 7f4e4cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:37:50.539 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.552+0000 7f4e4cff9640 1 -- 172.21.15.84:0/992632367 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1854747631 0 0) 0x7f4e501407e0 con 0x7f4e501403c0 2024-03-29T10:37:50.539 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.552+0000 7f4e4cff9640 1 -- 172.21.15.84:0/992632367 >> v1:172.21.15.86:6789/0 conn(0x7f4e500a9d70 legacy=0x7f4e501426f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:50.539 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.552+0000 7f4e4cff9640 1 -- 172.21.15.84:0/992632367 >> v1:172.21.15.84:6789/0 conn(0x7f4e5013f650 legacy=0x7f4e501da550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:37:50.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.555+0000 7f4e4cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:37:50.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.555+0000 7f4e4cff9640 1 monclient: found mon.c 2024-03-29T10:37:50.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.555+0000 7f4e4cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:50.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.555+0000 7f4e4cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:37:50.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.556+0000 7f4e4cff9640 1 -- 172.21.15.84:0/992632367 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e501db370 con 0x7f4e501403c0 2024-03-29T10:37:50.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.556+0000 7f4e4cff9640 10 monclient: _finish_auth 0 2024-03-29T10:37:50.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.556+0000 7f4e4cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:37:50.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.556+0000 7f4e4cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:50.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.556+0000 7f4e4cff9640 1 -- 172.21.15.84:0/992632367 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e48002010 con 0x7f4e501403c0 2024-03-29T10:37:50.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.556+0000 7f4e4cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:50.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.556+0000 7f4e4cff9640 1 -- 172.21.15.84:0/992632367 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4e48003de0 con 0x7f4e501403c0 2024-03-29T10:37:50.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.556+0000 7f4e4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:50.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.556+0000 7f4e4cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:37:50.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.556+0000 7f4e4cff9640 10 monclient: dump: 2024-03-29T10:37:50.542 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:50.542 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:50.542 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:50.542 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:50.543 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:50.543 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:50.543 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:50.543 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:50.543 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:50.543 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:50.543 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.556+0000 7f4e562fb640 5 monclient: authenticate success, global_id 4361 2024-03-29T10:37:50.543 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.556+0000 7f4e562fb640 10 monclient: _renew_subs 2024-03-29T10:37:50.543 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.556+0000 7f4e562fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:37:50.543 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.556+0000 7f4e562fb640 1 -- 172.21.15.84:0/992632367 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e5013b670 con 0x7f4e501403c0 2024-03-29T10:37:50.543 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.556+0000 7f4e562fb640 10 client.4361.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:37:50.543 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.557+0000 7f4e562fb640 10 monclient: _renew_subs 2024-03-29T10:37:50.543 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.557+0000 7f4e562fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:37:50.543 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.557+0000 7f4e562fb640 1 -- 172.21.15.84:0/992632367 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e5013b4b0 con 0x7f4e501403c0 2024-03-29T10:37:50.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.557+0000 7f4e562fb640 1 librados: init done 2024-03-29T10:37:50.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.557+0000 7f4e4e7fc640 4 set_mon_vals no callback set 2024-03-29T10:37:50.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.559+0000 7f4e562fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:50.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.559+0000 7f4e562fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:50.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.559+0000 7f4e562fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:37:50.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.559+0000 7f4e562fb640 1 -- 172.21.15.84:0/992632367 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e5013b860 con 0x7f4e501403c0 2024-03-29T10:37:50.547 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.562+0000 7f4e4cff9640 1 -- 172.21.15.84:0/992632367 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f4e48003f40 con 0x7f4e501403c0 2024-03-29T10:37:50.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.563+0000 7f4e4cff9640 1 -- 172.21.15.84:0/992632367 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f4e48084d50 con 0x7f4e501403c0 2024-03-29T10:37:50.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.563+0000 7f4e4cff9640 10 client.4361.objecter ms_dispatch 0x7f4e50000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:37:50.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.563+0000 7f4e4cff9640 3 client.4361.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:37:50.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.563+0000 7f4e4cff9640 3 client.4361.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:37:50.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.563+0000 7f4e4cff9640 20 client.4361.objecter dump_active .. 0 homeless 2024-03-29T10:37:50.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.563+0000 7f4e4ffff640 10 monclient: get_auth_request con 0x7f4e3c06dc50 auth_method 0 2024-03-29T10:37:50.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.564+0000 7f4e4cff9640 10 client.4361.objecter ms_handle_connect 0x7f4e3c06dc50 2024-03-29T10:37:50.556 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.572+0000 7f4e4cff9640 1 -- 172.21.15.84:0/992632367 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4e48050630 con 0x7f4e501403c0 2024-03-29T10:37:50.556 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.572+0000 7f4e4cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:50.556 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.572+0000 7f4e4cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:37:50.874 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.890+0000 7f4e562fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:50.874 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.890+0000 7f4e562fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:50.874 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.890+0000 7f4e562fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:37:50.874 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.890+0000 7f4e562fb640 1 -- 172.21.15.84:0/992632367 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e5013b4b0 con 0x7f4e501403c0 2024-03-29T10:37:50.875 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:37:50.876 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:37:50.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.891+0000 7f4e4cff9640 1 -- 172.21.15.84:0/992632367 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f4e5013b4b0 con 0x7f4e501403c0 2024-03-29T10:37:50.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.891+0000 7f4e4cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:50.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.891+0000 7f4e4cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:37:50.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.895+0000 7f4e562fb640 10 librados: watch_flush enter 2024-03-29T10:37:50.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.895+0000 7f4e562fb640 10 librados: watch_flush exit 2024-03-29T10:37:50.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.895+0000 7f4e562fb640 20 client.4361.objecter shutdown clearing up homeless session... 2024-03-29T10:37:50.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.895+0000 7f4e562fb640 10 client.4361.objecter successfully canceled tick 2024-03-29T10:37:50.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.896+0000 7f4e562fb640 1 -- 172.21.15.84:0/992632367 >> v1:172.21.15.86:6816/3337225337 conn(0x7f4e3c06dc50 legacy=0x7f4e3c070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:50.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.896+0000 7f4e562fb640 10 monclient: shutdown 2024-03-29T10:37:50.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.896+0000 7f4e562fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:50.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.896+0000 7f4e562fb640 1 -- 172.21.15.84:0/992632367 >> v1:172.21.15.84:6790/0 conn(0x7f4e501403c0 legacy=0x7f4e501dac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:50.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.896+0000 7f4e562fb640 1 -- 172.21.15.84:0/992632367 shutdown_connections 2024-03-29T10:37:50.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.896+0000 7f4e562fb640 1 -- 172.21.15.84:0/992632367 >> 172.21.15.84:0/992632367 conn(0x7f4e50066a00 msgr2=0x7f4e500b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:50.881 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.896+0000 7f4e562fb640 1 -- 172.21.15.84:0/992632367 shutdown_connections 2024-03-29T10:37:50.881 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.896+0000 7f4e562fb640 1 -- 172.21.15.84:0/992632367 wait complete. 2024-03-29T10:37:50.881 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:50.896+0000 7f4e562fb640 1 librados: shutdown 2024-03-29T10:37:50.897 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:37:57.704 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:37:57.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.871+0000 7f5ab4fcf640 10 monclient: get_monmap_and_config 2024-03-29T10:37:57.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.871+0000 7f5ab4fcf640 10 monclient: build_initial_monmap 2024-03-29T10:37:57.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.871+0000 7f5ab4fcf640 10 monclient: monmap: 2024-03-29T10:37:57.857 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:37:57.857 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:57.857 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:37:57.872644+0000 2024-03-29T10:37:57.857 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:37:57.872644+0000 2024-03-29T10:37:57.857 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:57.857 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:57.857 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:37:57.857 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:37:57.857 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:37:57.857 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:57.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.873+0000 7f5ab4fcf640 1 Processor -- start 2024-03-29T10:37:57.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.873+0000 7f5ab4fcf640 1 -- start start 2024-03-29T10:37:57.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.873+0000 7f5ab4fcf640 10 monclient: init 2024-03-29T10:37:57.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.874+0000 7f5ab4fcf640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:57.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.874+0000 7f5ab4fcf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:37:57.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.874+0000 7f5ab4fcf640 10 monclient(hunting): picked mon.noname-a con 0x7f5ab00a9ac0 addr v1:172.21.15.84:6789/0 2024-03-29T10:37:57.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.874+0000 7f5ab4fcf640 10 monclient(hunting): picked mon.noname-c con 0x7f5ab013ee90 addr v1:172.21.15.86:6789/0 2024-03-29T10:37:57.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.874+0000 7f5ab4fcf640 10 monclient(hunting): picked mon.noname-b con 0x7f5ab00af900 addr v1:172.21.15.84:6790/0 2024-03-29T10:37:57.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.874+0000 7f5ab4fcf640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab00afd10 con 0x7f5ab00a9ac0 2024-03-29T10:37:57.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.874+0000 7f5ab4fcf640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab00af6f0 con 0x7f5ab013ee90 2024-03-29T10:37:57.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.874+0000 7f5ab4fcf640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab00aac80 con 0x7f5ab00af900 2024-03-29T10:37:57.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.874+0000 7f5ab4fcf640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:57.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.874+0000 7f5ab4fcf640 10 monclient(hunting): authenticate will time out at 980.137534s 2024-03-29T10:37:57.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.880+0000 7f5aae59c640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f5ab00a9ac0 0x7f5ab00adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:41671/0 (socket says 172.21.15.84:41671) 2024-03-29T10:37:57.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.880+0000 7f5aae59c640 1 -- 172.21.15.84:0/1639620738 learned_addr learned my addr 172.21.15.84:0/1639620738 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:37:57.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.882+0000 7f5aad59a640 1 -- 172.21.15.84:0/1639620738 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435550708 0 0) 0x7f5ab00afd10 con 0x7f5ab00a9ac0 2024-03-29T10:37:57.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.882+0000 7f5aad59a640 10 monclient(hunting): _negotiate 2024-03-29T10:37:57.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.882+0000 7f5aad59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:57.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.882+0000 7f5aad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:57.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.882+0000 7f5aad59a640 10 monclient(hunting): my global_id is 4341 2024-03-29T10:37:57.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.882+0000 7f5aad59a640 1 -- 172.21.15.84:0/1639620738 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a98003500 con 0x7f5ab00a9ac0 2024-03-29T10:37:57.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.884+0000 7f5aad59a640 1 -- 172.21.15.84:0/1639620738 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773061050 0 0) 0x7f5ab00af6f0 con 0x7f5ab013ee90 2024-03-29T10:37:57.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.884+0000 7f5aad59a640 10 monclient(hunting): _negotiate 2024-03-29T10:37:57.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.884+0000 7f5aad59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:57.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.884+0000 7f5aad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:57.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.884+0000 7f5aad59a640 10 monclient(hunting): my global_id is 4330 2024-03-29T10:37:57.869 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.884+0000 7f5aad59a640 1 -- 172.21.15.84:0/1639620738 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ab00afd10 con 0x7f5ab013ee90 2024-03-29T10:37:57.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.885+0000 7f5aad59a640 1 -- 172.21.15.84:0/1639620738 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1094513589 0 0) 0x7f5ab00aac80 con 0x7f5ab00af900 2024-03-29T10:37:57.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.885+0000 7f5aad59a640 10 monclient(hunting): _negotiate 2024-03-29T10:37:57.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.885+0000 7f5aad59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:57.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.885+0000 7f5aad59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:57.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.885+0000 7f5aad59a640 10 monclient(hunting): my global_id is 4364 2024-03-29T10:37:57.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.885+0000 7f5aad59a640 1 -- 172.21.15.84:0/1639620738 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ab00af6f0 con 0x7f5ab00af900 2024-03-29T10:37:57.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.885+0000 7f5aad59a640 1 -- 172.21.15.84:0/1639620738 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2619418497 0 0) 0x7f5a98003500 con 0x7f5ab00a9ac0 2024-03-29T10:37:57.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.885+0000 7f5aad59a640 1 -- 172.21.15.84:0/1639620738 >> v1:172.21.15.84:6790/0 conn(0x7f5ab00af900 legacy=0x7f5ab0149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:57.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.885+0000 7f5aad59a640 1 -- 172.21.15.84:0/1639620738 >> v1:172.21.15.86:6789/0 conn(0x7f5ab013ee90 legacy=0x7f5ab013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:57.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.885+0000 7f5aad59a640 10 monclient: _finish_hunting 0 2024-03-29T10:37:57.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.885+0000 7f5aad59a640 1 monclient: found mon.noname-a 2024-03-29T10:37:57.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.885+0000 7f5aad59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:57.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.885+0000 7f5aad59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T10:37:57.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.885+0000 7f5aad59a640 1 -- 172.21.15.84:0/1639620738 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab00aaf60 con 0x7f5ab00a9ac0 2024-03-29T10:37:57.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.885+0000 7f5aad59a640 10 monclient: _finish_auth 0 2024-03-29T10:37:57.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.885+0000 7f5aad59a640 10 monclient: _check_auth_tickets 2024-03-29T10:37:57.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.885+0000 7f5aad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:57.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.885+0000 7f5aad59a640 1 -- 172.21.15.84:0/1639620738 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f5aa0004cb0 con 0x7f5ab00a9ac0 2024-03-29T10:37:57.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.885+0000 7f5aad59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:57.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.885+0000 7f5aad59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:37:57.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.885+0000 7f5aad59a640 10 monclient: dump: 2024-03-29T10:37:57.871 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:57.871 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:57.871 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:57.871 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:57.871 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:57.872 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:57.872 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:57.872 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:57.872 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:57.872 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:57.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.885+0000 7f5ab4fcf640 5 monclient: authenticate success, global_id 4341 2024-03-29T10:37:57.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.885+0000 7f5ab4fcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:37:57.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.885+0000 7f5aad59a640 1 -- 172.21.15.84:0/1639620738 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5aa0004e10 con 0x7f5ab00a9ac0 2024-03-29T10:37:57.872 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.885+0000 7f5aad59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:57.873 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.885+0000 7f5ab4fcf640 10 monclient: get_monmap_and_config success 2024-03-29T10:37:57.873 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.885+0000 7f5ab4fcf640 4 set_mon_vals no callback set 2024-03-29T10:37:57.873 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.886+0000 7f5ab4fcf640 10 monclient: shutdown 2024-03-29T10:37:57.873 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.886+0000 7f5ab4fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:57.873 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.886+0000 7f5ab4fcf640 1 -- 172.21.15.84:0/1639620738 >> v1:172.21.15.84:6789/0 conn(0x7f5ab00a9ac0 legacy=0x7f5ab00adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:57.874 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.889+0000 7f5ab4fcf640 1 -- 172.21.15.84:0/1639620738 shutdown_connections 2024-03-29T10:37:57.874 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.889+0000 7f5ab4fcf640 1 -- 172.21.15.84:0/1639620738 >> 172.21.15.84:0/1639620738 conn(0x7f5ab00b35e0 msgr2=0x7f5ab00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:57.874 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.890+0000 7f5ab4fcf640 1 -- 172.21.15.84:0/1639620738 shutdown_connections 2024-03-29T10:37:57.874 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.890+0000 7f5ab4fcf640 1 -- 172.21.15.84:0/1639620738 wait complete. 2024-03-29T10:37:57.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.891+0000 7f5ab4fcf640 10 monclient: build_initial_monmap 2024-03-29T10:37:57.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.891+0000 7f5ab4fcf640 10 monclient: monmap: 2024-03-29T10:37:57.876 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:37:57.876 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:37:57.876 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:37:57.876 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:37:57.876 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:37:57.876 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:57.877 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:37:57.877 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:37:57.877 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:37:57.877 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:57.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.892+0000 7f5ab4fcf640 1 librados: starting msgr at 2024-03-29T10:37:57.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.892+0000 7f5ab4fcf640 1 librados: starting objecter 2024-03-29T10:37:57.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.892+0000 7f5ab4fcf640 1 Processor -- start 2024-03-29T10:37:57.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.892+0000 7f5ab4fcf640 1 -- start start 2024-03-29T10:37:57.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.892+0000 7f5ab4fcf640 1 librados: setting wanted keys 2024-03-29T10:37:57.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.892+0000 7f5ab4fcf640 1 librados: calling monclient init 2024-03-29T10:37:57.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.892+0000 7f5ab4fcf640 10 monclient: init 2024-03-29T10:37:57.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.893+0000 7f5ab4fcf640 10 monclient: _reopen_session rank -1 2024-03-29T10:37:57.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.893+0000 7f5ab4fcf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:37:57.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.893+0000 7f5ab4fcf640 10 monclient(hunting): picked mon.noname-b con 0x7f5ab00a9ac0 addr v1:172.21.15.86:6789/0 2024-03-29T10:37:57.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.893+0000 7f5ab4fcf640 10 monclient(hunting): picked mon.noname-a con 0x7f5ab00af900 addr v1:172.21.15.84:6789/0 2024-03-29T10:37:57.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.893+0000 7f5ab4fcf640 10 monclient(hunting): picked mon.noname-c con 0x7f5ab013ee90 addr v1:172.21.15.84:6790/0 2024-03-29T10:37:57.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.893+0000 7f5ab4fcf640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab00701a0 con 0x7f5ab00af900 2024-03-29T10:37:57.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.893+0000 7f5ab4fcf640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab0071340 con 0x7f5ab00a9ac0 2024-03-29T10:37:57.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.893+0000 7f5ab4fcf640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ab0071620 con 0x7f5ab013ee90 2024-03-29T10:37:57.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.893+0000 7f5ab4fcf640 10 monclient(hunting): _renew_subs 2024-03-29T10:37:57.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.893+0000 7f5ab4fcf640 10 monclient(hunting): authenticate will time out at 980.156329s 2024-03-29T10:37:57.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.899+0000 7f5aaed9d640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f5ab013ee90 0x7f5ab01ef8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:42663/0 (socket says 172.21.15.84:42663) 2024-03-29T10:37:57.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.899+0000 7f5aaed9d640 1 -- 172.21.15.84:0/3179384394 learned_addr learned my addr 172.21.15.84:0/3179384394 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:37:57.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.899+0000 7f5a9effd640 10 client.?.objecter ms_handle_connect 0x7f5ab013ee90 2024-03-29T10:37:57.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.899+0000 7f5a9effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:57.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.899+0000 7f5a9effd640 1 -- 172.21.15.84:0/3179384394 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013073363 0 0) 0x7f5ab0071620 con 0x7f5ab013ee90 2024-03-29T10:37:57.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.899+0000 7f5a9effd640 10 monclient(hunting): _negotiate 2024-03-29T10:37:57.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.899+0000 7f5a9effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:57.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.899+0000 7f5a9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:57.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.899+0000 7f5a9effd640 10 monclient(hunting): my global_id is 4367 2024-03-29T10:37:57.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.899+0000 7f5a9effd640 1 -- 172.21.15.84:0/3179384394 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a90004eb0 con 0x7f5ab013ee90 2024-03-29T10:37:57.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.900+0000 7f5a9effd640 1 -- 172.21.15.84:0/3179384394 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 346905505 0 0) 0x7f5a90004eb0 con 0x7f5ab013ee90 2024-03-29T10:37:57.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.900+0000 7f5a9effd640 1 -- 172.21.15.84:0/3179384394 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ab0071620 con 0x7f5ab013ee90 2024-03-29T10:37:57.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.900+0000 7f5a9effd640 1 -- 172.21.15.84:0/3179384394 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f5aa4001990 con 0x7f5ab013ee90 2024-03-29T10:37:57.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.900+0000 7f5a9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:57.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.900+0000 7f5a9effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:37:57.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.900+0000 7f5a9effd640 10 monclient(hunting): dump: 2024-03-29T10:37:57.886 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:57.887 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:57.887 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:57.887 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:57.887 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:57.887 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:57.887 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:57.887 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:57.887 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:57.887 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:57.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.900+0000 7f5a9effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:37:57.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.900+0000 7f5a9effd640 10 client.?.objecter ms_handle_connect 0x7f5ab00af900 2024-03-29T10:37:57.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.900+0000 7f5a9effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:57.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.901+0000 7f5a9effd640 1 -- 172.21.15.84:0/3179384394 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 588736321 0 0) 0x7f5ab00701a0 con 0x7f5ab00af900 2024-03-29T10:37:57.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.901+0000 7f5a9effd640 10 monclient(hunting): _negotiate 2024-03-29T10:37:57.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.901+0000 7f5a9effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:37:57.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.901+0000 7f5a9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:37:57.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.901+0000 7f5a9effd640 10 monclient(hunting): my global_id is 4344 2024-03-29T10:37:57.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.901+0000 7f5a9effd640 1 -- 172.21.15.84:0/3179384394 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a90004eb0 con 0x7f5ab00af900 2024-03-29T10:37:57.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.901+0000 7f5a9effd640 1 -- 172.21.15.84:0/3179384394 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2548045066 0 0) 0x7f5a90004eb0 con 0x7f5ab00af900 2024-03-29T10:37:57.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.901+0000 7f5a9effd640 1 -- 172.21.15.84:0/3179384394 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ab00701a0 con 0x7f5ab00af900 2024-03-29T10:37:57.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.901+0000 7f5a9effd640 1 -- 172.21.15.84:0/3179384394 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f5aa8001bd0 con 0x7f5ab00af900 2024-03-29T10:37:57.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.901+0000 7f5a9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:57.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.901+0000 7f5a9effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:37:57.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.901+0000 7f5a9effd640 10 monclient(hunting): dump: 2024-03-29T10:37:57.888 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:57.888 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:57.888 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:57.888 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:57.888 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:57.888 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:57.889 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:57.889 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:57.889 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:57.889 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:57.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.904+0000 7f5a9effd640 10 client.?.objecter ms_handle_connect 0x7f5ab00a9ac0 2024-03-29T10:37:57.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.904+0000 7f5a9effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:37:57.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.904+0000 7f5a9effd640 1 -- 172.21.15.84:0/3179384394 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1509653242 0 0) 0x7f5ab00701a0 con 0x7f5ab00af900 2024-03-29T10:37:57.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.904+0000 7f5a9effd640 1 -- 172.21.15.84:0/3179384394 >> v1:172.21.15.84:6790/0 conn(0x7f5ab013ee90 legacy=0x7f5ab01ef8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:57.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.904+0000 7f5a9effd640 1 -- 172.21.15.84:0/3179384394 >> v1:172.21.15.86:6789/0 conn(0x7f5ab00a9ac0 legacy=0x7f5ab01e4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:57.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.904+0000 7f5a9effd640 10 monclient: _finish_hunting 0 2024-03-29T10:37:57.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.904+0000 7f5a9effd640 1 monclient: found mon.a 2024-03-29T10:37:57.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.904+0000 7f5a9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:37:57.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.904+0000 7f5a9effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:37:57.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.904+0000 7f5a9effd640 1 -- 172.21.15.84:0/3179384394 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab01e4780 con 0x7f5ab00af900 2024-03-29T10:37:57.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.904+0000 7f5a9effd640 10 monclient: _finish_auth 0 2024-03-29T10:37:57.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.904+0000 7f5a9effd640 10 monclient: _check_auth_tickets 2024-03-29T10:37:57.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.904+0000 7f5a9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:37:57.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.904+0000 7f5ab4fcf640 5 monclient: authenticate success, global_id 4344 2024-03-29T10:37:57.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.904+0000 7f5ab4fcf640 10 monclient: _renew_subs 2024-03-29T10:37:57.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.904+0000 7f5ab4fcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:37:57.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.904+0000 7f5ab4fcf640 1 -- 172.21.15.84:0/3179384394 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab0070630 con 0x7f5ab00af900 2024-03-29T10:37:57.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.904+0000 7f5ab4fcf640 10 client.4344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:37:57.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.904+0000 7f5ab4fcf640 10 monclient: _renew_subs 2024-03-29T10:37:57.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.904+0000 7f5ab4fcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:37:57.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.904+0000 7f5ab4fcf640 1 -- 172.21.15.84:0/3179384394 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab0072270 con 0x7f5ab00af900 2024-03-29T10:37:57.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.904+0000 7f5ab4fcf640 1 librados: init done 2024-03-29T10:37:57.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.904+0000 7f5a9effd640 1 -- 172.21.15.84:0/3179384394 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5aa8003970 con 0x7f5ab00af900 2024-03-29T10:37:57.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.904+0000 7f5a9effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:37:57.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.904+0000 7f5a9effd640 1 -- 172.21.15.84:0/3179384394 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f5aa8003c80 con 0x7f5ab00af900 2024-03-29T10:37:57.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.904+0000 7f5a9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:37:57.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.904+0000 7f5a9effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:37:57.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.904+0000 7f5a9effd640 10 monclient: dump: 2024-03-29T10:37:57.891 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:37:57.892 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:37:57.892 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:57.892 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:37:57.892 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:37:57.892 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:37:57.892 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:37:57.892 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:37:57.892 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:37:57.892 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:37:57.892 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.904+0000 7f5a9ffff640 4 set_mon_vals no callback set 2024-03-29T10:37:57.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.908+0000 7f5ab4fcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:57.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.908+0000 7f5ab4fcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:57.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.908+0000 7f5ab4fcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:37:57.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.908+0000 7f5ab4fcf640 1 -- 172.21.15.84:0/3179384394 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab013f3b0 con 0x7f5ab00af900 2024-03-29T10:37:57.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.910+0000 7f5a9effd640 1 -- 172.21.15.84:0/3179384394 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f5aa8019c50 con 0x7f5ab00af900 2024-03-29T10:37:57.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.910+0000 7f5a9effd640 1 -- 172.21.15.84:0/3179384394 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f5aa8083aa0 con 0x7f5ab00af900 2024-03-29T10:37:57.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.910+0000 7f5a9effd640 10 client.4344.objecter ms_dispatch 0x7f5ab0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:37:57.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.910+0000 7f5a9effd640 3 client.4344.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:37:57.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.910+0000 7f5a9effd640 3 client.4344.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:37:57.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.910+0000 7f5a9effd640 20 client.4344.objecter dump_active .. 0 homeless 2024-03-29T10:37:57.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.911+0000 7f5aae59c640 10 monclient: get_auth_request con 0x7f5a9006deb0 auth_method 0 2024-03-29T10:37:57.898 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.914+0000 7f5a9effd640 10 client.4344.objecter ms_handle_connect 0x7f5a9006deb0 2024-03-29T10:37:57.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.918+0000 7f5a9effd640 1 -- 172.21.15.84:0/3179384394 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5aa804f380 con 0x7f5ab00af900 2024-03-29T10:37:57.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.918+0000 7f5a9effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:37:57.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:57.918+0000 7f5a9effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:37:58.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:58.244+0000 7f5ab4fcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:58.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:58.244+0000 7f5ab4fcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:58.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:58.244+0000 7f5ab4fcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:37:58.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:58.244+0000 7f5ab4fcf640 1 -- 172.21.15.84:0/3179384394 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ab0070330 con 0x7f5ab00af900 2024-03-29T10:37:58.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:58.245+0000 7f5a9effd640 1 -- 172.21.15.84:0/3179384394 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f5aa80512b0 con 0x7f5ab00af900 2024-03-29T10:37:58.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:58.245+0000 7f5a9effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:37:58.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:58.245+0000 7f5a9effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:37:58.231 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:37:58.231 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:37:58.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:58.248+0000 7f5ab4fcf640 10 librados: watch_flush enter 2024-03-29T10:37:58.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:58.248+0000 7f5ab4fcf640 10 librados: watch_flush exit 2024-03-29T10:37:58.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:58.248+0000 7f5ab4fcf640 20 client.4344.objecter shutdown clearing up homeless session... 2024-03-29T10:37:58.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:58.248+0000 7f5ab4fcf640 10 client.4344.objecter successfully canceled tick 2024-03-29T10:37:58.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:58.249+0000 7f5ab4fcf640 1 -- 172.21.15.84:0/3179384394 >> v1:172.21.15.86:6816/3337225337 conn(0x7f5a9006deb0 legacy=0x7f5a90070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:58.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:58.249+0000 7f5ab4fcf640 10 monclient: shutdown 2024-03-29T10:37:58.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:58.249+0000 7f5ab4fcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:37:58.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:58.249+0000 7f5ab4fcf640 1 -- 172.21.15.84:0/3179384394 >> v1:172.21.15.84:6789/0 conn(0x7f5ab00af900 legacy=0x7f5ab01e1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:37:58.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:58.249+0000 7f5ab4fcf640 1 -- 172.21.15.84:0/3179384394 shutdown_connections 2024-03-29T10:37:58.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:58.249+0000 7f5ab4fcf640 1 -- 172.21.15.84:0/3179384394 >> 172.21.15.84:0/3179384394 conn(0x7f5ab00b35e0 msgr2=0x7f5ab00ad140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:37:58.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:58.249+0000 7f5ab4fcf640 1 -- 172.21.15.84:0/3179384394 shutdown_connections 2024-03-29T10:37:58.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:58.249+0000 7f5ab4fcf640 1 -- 172.21.15.84:0/3179384394 wait complete. 2024-03-29T10:37:58.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:37:58.249+0000 7f5ab4fcf640 1 librados: shutdown 2024-03-29T10:37:58.251 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:38:05.059 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:38:05.209 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.225+0000 7f2ac2b75640 10 monclient: get_monmap_and_config 2024-03-29T10:38:05.209 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.225+0000 7f2ac2b75640 10 monclient: build_initial_monmap 2024-03-29T10:38:05.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.225+0000 7f2ac2b75640 10 monclient: monmap: 2024-03-29T10:38:05.210 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:38:05.210 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:05.210 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:38:05.226262+0000 2024-03-29T10:38:05.210 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:38:05.226262+0000 2024-03-29T10:38:05.210 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:05.210 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:05.210 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:38:05.210 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:38:05.210 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:38:05.210 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:05.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.228+0000 7f2ac2b75640 1 Processor -- start 2024-03-29T10:38:05.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.230+0000 7f2ac2b75640 1 -- start start 2024-03-29T10:38:05.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.230+0000 7f2ac2b75640 10 monclient: init 2024-03-29T10:38:05.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.231+0000 7f2ac2b75640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:05.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.231+0000 7f2ac2b75640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:38:05.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.231+0000 7f2ac2b75640 10 monclient(hunting): picked mon.noname-a con 0x7f2abc1499d0 addr v1:172.21.15.84:6789/0 2024-03-29T10:38:05.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.231+0000 7f2ac2b75640 10 monclient(hunting): picked mon.noname-c con 0x7f2abc0a9fc0 addr v1:172.21.15.86:6789/0 2024-03-29T10:38:05.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.231+0000 7f2ac2b75640 10 monclient(hunting): picked mon.noname-b con 0x7f2abc14faf0 addr v1:172.21.15.84:6790/0 2024-03-29T10:38:05.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.231+0000 7f2ac2b75640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2abc0abd70 con 0x7f2abc1499d0 2024-03-29T10:38:05.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.231+0000 7f2ac2b75640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2abc0a9850 con 0x7f2abc0a9fc0 2024-03-29T10:38:05.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.231+0000 7f2ac2b75640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2abc1405c0 con 0x7f2abc14faf0 2024-03-29T10:38:05.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.231+0000 7f2ac2b75640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:05.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.231+0000 7f2ac2b75640 10 monclient(hunting): authenticate will time out at 987.494288s 2024-03-29T10:38:05.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.234+0000 7f2ac1372640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f2abc0a9fc0 0x7f2abc0aa420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:52411/0 (socket says 172.21.15.84:52411) 2024-03-29T10:38:05.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.234+0000 7f2ac1372640 1 -- 172.21.15.84:0/3060326693 learned_addr learned my addr 172.21.15.84:0/3060326693 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:38:05.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.234+0000 7f2ac0b71640 1 -- 172.21.15.84:0/3060326693 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2756650003 0 0) 0x7f2abc0a9850 con 0x7f2abc0a9fc0 2024-03-29T10:38:05.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.234+0000 7f2ac0b71640 10 monclient(hunting): _negotiate 2024-03-29T10:38:05.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.234+0000 7f2ac0b71640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:05.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.234+0000 7f2ac0b71640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:05.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.234+0000 7f2ac0b71640 10 monclient(hunting): my global_id is 4336 2024-03-29T10:38:05.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.234+0000 7f2ac0b71640 1 -- 172.21.15.84:0/3060326693 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aa8003520 con 0x7f2abc0a9fc0 2024-03-29T10:38:05.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.235+0000 7f2ac0b71640 1 -- 172.21.15.84:0/3060326693 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 272780714 0 0) 0x7f2aa8003520 con 0x7f2abc0a9fc0 2024-03-29T10:38:05.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.235+0000 7f2ac0b71640 1 -- 172.21.15.84:0/3060326693 >> v1:172.21.15.84:6790/0 conn(0x7f2abc14faf0 legacy=0x7f2abc151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:05.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.235+0000 7f2ac0b71640 1 -- 172.21.15.84:0/3060326693 >> v1:172.21.15.84:6789/0 conn(0x7f2abc1499d0 legacy=0x7f2abc0ad820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:05.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.235+0000 7f2ac0b71640 10 monclient: _finish_hunting 0 2024-03-29T10:38:05.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.235+0000 7f2ac0b71640 1 monclient: found mon.noname-c 2024-03-29T10:38:05.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.235+0000 7f2ac0b71640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:05.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.235+0000 7f2ac0b71640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-29T10:38:05.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.235+0000 7f2ac0b71640 1 -- 172.21.15.84:0/3060326693 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2abc143d20 con 0x7f2abc0a9fc0 2024-03-29T10:38:05.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.235+0000 7f2ac0b71640 10 monclient: _finish_auth 0 2024-03-29T10:38:05.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.235+0000 7f2ac0b71640 10 monclient: _check_auth_tickets 2024-03-29T10:38:05.222 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.235+0000 7f2ac0b71640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:05.222 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.235+0000 7f2ac2b75640 5 monclient: authenticate success, global_id 4336 2024-03-29T10:38:05.222 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.235+0000 7f2ac2b75640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:05.222 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.235+0000 7f2ac0b71640 1 -- 172.21.15.84:0/3060326693 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f2ab4001a10 con 0x7f2abc0a9fc0 2024-03-29T10:38:05.222 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.235+0000 7f2ac0b71640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:05.222 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.235+0000 7f2ac0b71640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:38:05.222 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.235+0000 7f2ac0b71640 10 monclient: dump: 2024-03-29T10:38:05.222 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:05.222 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:05.222 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:05.222 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:05.222 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:05.223 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:05.223 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:05.223 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:05.223 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:05.223 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:05.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.235+0000 7f2ac2b75640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:05.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.235+0000 7f2ac0b71640 1 -- 172.21.15.84:0/3060326693 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ab4001c30 con 0x7f2abc0a9fc0 2024-03-29T10:38:05.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.235+0000 7f2ac0b71640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:05.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.235+0000 7f2ac2b75640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:05.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.235+0000 7f2ac2b75640 4 set_mon_vals no callback set 2024-03-29T10:38:05.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.236+0000 7f2ac2b75640 10 monclient: shutdown 2024-03-29T10:38:05.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.236+0000 7f2ac2b75640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:05.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.236+0000 7f2ac2b75640 1 -- 172.21.15.84:0/3060326693 >> v1:172.21.15.86:6789/0 conn(0x7f2abc0a9fc0 legacy=0x7f2abc0aa420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:05.226 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.238+0000 7f2ac2b75640 1 -- 172.21.15.84:0/3060326693 shutdown_connections 2024-03-29T10:38:05.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.238+0000 7f2ac2b75640 1 -- 172.21.15.84:0/3060326693 >> 172.21.15.84:0/3060326693 conn(0x7f2abc0b3c50 msgr2=0x7f2abc0b4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:05.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.238+0000 7f2ac2b75640 1 -- 172.21.15.84:0/3060326693 shutdown_connections 2024-03-29T10:38:05.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.239+0000 7f2ac2b75640 1 -- 172.21.15.84:0/3060326693 wait complete. 2024-03-29T10:38:05.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.239+0000 7f2ac2b75640 10 monclient: build_initial_monmap 2024-03-29T10:38:05.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.239+0000 7f2ac2b75640 10 monclient: monmap: 2024-03-29T10:38:05.227 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:38:05.227 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:05.227 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:38:05.227 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:38:05.227 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:05.227 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:05.227 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:38:05.228 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:38:05.228 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:38:05.228 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:05.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.240+0000 7f2ac2b75640 1 librados: starting msgr at 2024-03-29T10:38:05.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.240+0000 7f2ac2b75640 1 librados: starting objecter 2024-03-29T10:38:05.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.240+0000 7f2ac2b75640 1 Processor -- start 2024-03-29T10:38:05.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.240+0000 7f2ac2b75640 1 -- start start 2024-03-29T10:38:05.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.240+0000 7f2ac2b75640 1 librados: setting wanted keys 2024-03-29T10:38:05.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.240+0000 7f2ac2b75640 1 librados: calling monclient init 2024-03-29T10:38:05.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.240+0000 7f2ac2b75640 10 monclient: init 2024-03-29T10:38:05.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.241+0000 7f2ac2b75640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:05.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.241+0000 7f2ac2b75640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:38:05.228 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.241+0000 7f2ac2b75640 10 monclient(hunting): picked mon.noname-c con 0x7f2abc0a9fc0 addr v1:172.21.15.84:6790/0 2024-03-29T10:38:05.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.241+0000 7f2ac2b75640 10 monclient(hunting): picked mon.noname-a con 0x7f2abc1499d0 addr v1:172.21.15.84:6789/0 2024-03-29T10:38:05.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.241+0000 7f2ac2b75640 10 monclient(hunting): picked mon.noname-b con 0x7f2abc14faf0 addr v1:172.21.15.86:6789/0 2024-03-29T10:38:05.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.241+0000 7f2ac2b75640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2abc142ba0 con 0x7f2abc1499d0 2024-03-29T10:38:05.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.241+0000 7f2ac2b75640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2abc1428b0 con 0x7f2abc14faf0 2024-03-29T10:38:05.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.241+0000 7f2ac2b75640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2abc141f80 con 0x7f2abc0a9fc0 2024-03-29T10:38:05.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.241+0000 7f2ac2b75640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:05.229 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.241+0000 7f2ac2b75640 10 monclient(hunting): authenticate will time out at 987.504545s 2024-03-29T10:38:05.231 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.244+0000 7f2ac1b73640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f2abc0a9fc0 0x7f2abc13c5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:48646/0 (socket says 172.21.15.84:48646) 2024-03-29T10:38:05.231 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.244+0000 7f2ac1b73640 1 -- 172.21.15.84:0/2317046924 learned_addr learned my addr 172.21.15.84:0/2317046924 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:38:05.231 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.244+0000 7f2ab27fc640 10 client.?.objecter ms_handle_connect 0x7f2abc14faf0 2024-03-29T10:38:05.231 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.244+0000 7f2ab27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:05.231 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.244+0000 7f2ab27fc640 10 client.?.objecter ms_handle_connect 0x7f2abc0a9fc0 2024-03-29T10:38:05.231 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.245+0000 7f2ab27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:05.231 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.245+0000 7f2ab27fc640 1 -- 172.21.15.84:0/2317046924 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 67869609 0 0) 0x7f2abc1428b0 con 0x7f2abc14faf0 2024-03-29T10:38:05.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.245+0000 7f2ab27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:05.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.245+0000 7f2ab27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:05.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.245+0000 7f2ab27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:05.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.245+0000 7f2ab27fc640 10 monclient(hunting): my global_id is 4339 2024-03-29T10:38:05.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.245+0000 7f2ab27fc640 1 -- 172.21.15.84:0/2317046924 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aa0004eb0 con 0x7f2abc14faf0 2024-03-29T10:38:05.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.245+0000 7f2ab27fc640 1 -- 172.21.15.84:0/2317046924 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 781981661 0 0) 0x7f2aa0004eb0 con 0x7f2abc14faf0 2024-03-29T10:38:05.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.245+0000 7f2ab27fc640 1 -- 172.21.15.84:0/2317046924 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2abc1428b0 con 0x7f2abc14faf0 2024-03-29T10:38:05.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.245+0000 7f2ab27fc640 1 -- 172.21.15.84:0/2317046924 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f2aac001810 con 0x7f2abc14faf0 2024-03-29T10:38:05.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.245+0000 7f2ab27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:05.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.245+0000 7f2ab27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:38:05.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.245+0000 7f2ab27fc640 10 monclient(hunting): dump: 2024-03-29T10:38:05.233 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:05.233 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:05.233 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:05.233 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:05.233 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:05.233 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:05.233 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:05.233 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:05.233 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:05.233 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:05.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.245+0000 7f2ab27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:05.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.245+0000 7f2ab27fc640 1 -- 172.21.15.84:0/2317046924 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3502612136 0 0) 0x7f2abc141f80 con 0x7f2abc0a9fc0 2024-03-29T10:38:05.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.246+0000 7f2ab27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:05.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.246+0000 7f2ab27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:05.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.246+0000 7f2ab27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:05.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.246+0000 7f2ab27fc640 10 monclient(hunting): my global_id is 4373 2024-03-29T10:38:05.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.246+0000 7f2ab27fc640 1 -- 172.21.15.84:0/2317046924 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aa0004eb0 con 0x7f2abc0a9fc0 2024-03-29T10:38:05.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.246+0000 7f2ab27fc640 1 -- 172.21.15.84:0/2317046924 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1037874485 0 0) 0x7f2abc1428b0 con 0x7f2abc14faf0 2024-03-29T10:38:05.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.246+0000 7f2ab27fc640 1 -- 172.21.15.84:0/2317046924 >> v1:172.21.15.84:6790/0 conn(0x7f2abc0a9fc0 legacy=0x7f2abc13c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:05.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.246+0000 7f2ab27fc640 1 -- 172.21.15.84:0/2317046924 >> v1:172.21.15.84:6789/0 conn(0x7f2abc1499d0 legacy=0x7f2abc1de7f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:38:05.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.248+0000 7f2ab27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:38:05.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.248+0000 7f2ab27fc640 1 monclient: found mon.b 2024-03-29T10:38:05.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.248+0000 7f2ab27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:05.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.248+0000 7f2ab27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:38:05.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.248+0000 7f2ab27fc640 1 -- 172.21.15.84:0/2317046924 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2abc142110 con 0x7f2abc14faf0 2024-03-29T10:38:05.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.248+0000 7f2ab27fc640 10 monclient: _finish_auth 0 2024-03-29T10:38:05.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.248+0000 7f2ab27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:38:05.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.248+0000 7f2ab27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:05.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.248+0000 7f2ac2b75640 5 monclient: authenticate success, global_id 4339 2024-03-29T10:38:05.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.248+0000 7f2ac2b75640 10 monclient: _renew_subs 2024-03-29T10:38:05.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.248+0000 7f2ac2b75640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:38:05.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.248+0000 7f2ac2b75640 1 -- 172.21.15.84:0/2317046924 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2abc141aa0 con 0x7f2abc14faf0 2024-03-29T10:38:05.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.249+0000 7f2ac2b75640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:05.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.249+0000 7f2ac2b75640 10 monclient: _renew_subs 2024-03-29T10:38:05.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.249+0000 7f2ac2b75640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:38:05.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.249+0000 7f2ac2b75640 1 -- 172.21.15.84:0/2317046924 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2abc145f70 con 0x7f2abc14faf0 2024-03-29T10:38:05.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.249+0000 7f2ac2b75640 1 librados: init done 2024-03-29T10:38:05.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.250+0000 7f2ac2b75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:05.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.250+0000 7f2ac2b75640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:05.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.250+0000 7f2ac2b75640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:38:05.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.250+0000 7f2ac2b75640 1 -- 172.21.15.84:0/2317046924 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2abc04f730 con 0x7f2abc14faf0 2024-03-29T10:38:05.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.251+0000 7f2ab27fc640 1 -- 172.21.15.84:0/2317046924 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2aac001970 con 0x7f2abc14faf0 2024-03-29T10:38:05.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.251+0000 7f2ab27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:05.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.251+0000 7f2ab27fc640 1 -- 172.21.15.84:0/2317046924 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f2aac0037b0 con 0x7f2abc14faf0 2024-03-29T10:38:05.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.251+0000 7f2ab27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:05.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.251+0000 7f2ab27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:05.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.252+0000 7f2ab3fff640 4 set_mon_vals no callback set 2024-03-29T10:38:05.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.252+0000 7f2ab27fc640 10 monclient: dump: 2024-03-29T10:38:05.239 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:05.239 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:05.239 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:05.239 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:05.240 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:05.240 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:05.240 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:05.240 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:05.240 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:05.240 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:05.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.253+0000 7f2ab27fc640 1 -- 172.21.15.84:0/2317046924 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f2aac003a10 con 0x7f2abc14faf0 2024-03-29T10:38:05.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.254+0000 7f2ab27fc640 1 -- 172.21.15.84:0/2317046924 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f2aac0837f0 con 0x7f2abc14faf0 2024-03-29T10:38:05.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.254+0000 7f2ab27fc640 10 client.4339.objecter ms_dispatch 0x7f2abc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:38:05.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.254+0000 7f2ab27fc640 3 client.4339.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:38:05.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.254+0000 7f2ab27fc640 3 client.4339.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:38:05.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.254+0000 7f2ab27fc640 20 client.4339.objecter dump_active .. 0 homeless 2024-03-29T10:38:05.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.255+0000 7f2ac1b73640 10 monclient: get_auth_request con 0x7f2aa006dc90 auth_method 0 2024-03-29T10:38:05.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.256+0000 7f2ab27fc640 10 client.4339.objecter ms_handle_connect 0x7f2aa006dc90 2024-03-29T10:38:05.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.257+0000 7f2ab27fc640 1 -- 172.21.15.84:0/2317046924 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2aac04f0d0 con 0x7f2abc14faf0 2024-03-29T10:38:05.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.258+0000 7f2ab27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:05.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.258+0000 7f2ab27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:05.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.612+0000 7f2ac2b75640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:05.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.612+0000 7f2ac2b75640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:05.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.612+0000 7f2ac2b75640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:38:05.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.612+0000 7f2ac2b75640 1 -- 172.21.15.84:0/2317046924 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2abc142d30 con 0x7f2abc14faf0 2024-03-29T10:38:05.599 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.614+0000 7f2ab27fc640 1 -- 172.21.15.84:0/2317046924 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f2aac051000 con 0x7f2abc14faf0 2024-03-29T10:38:05.599 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.614+0000 7f2ab27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:05.599 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.614+0000 7f2ab27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:05.599 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:38:05.599 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:38:05.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.619+0000 7f2ac2b75640 10 librados: watch_flush enter 2024-03-29T10:38:05.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.619+0000 7f2ac2b75640 10 librados: watch_flush exit 2024-03-29T10:38:05.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.619+0000 7f2ac2b75640 20 client.4339.objecter shutdown clearing up homeless session... 2024-03-29T10:38:05.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.619+0000 7f2ac2b75640 10 client.4339.objecter successfully canceled tick 2024-03-29T10:38:05.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.620+0000 7f2ac2b75640 1 -- 172.21.15.84:0/2317046924 >> v1:172.21.15.86:6816/3337225337 conn(0x7f2aa006dc90 legacy=0x7f2aa0070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:05.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.620+0000 7f2ac2b75640 10 monclient: shutdown 2024-03-29T10:38:05.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.620+0000 7f2ac2b75640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:05.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.620+0000 7f2ac2b75640 1 -- 172.21.15.84:0/2317046924 >> v1:172.21.15.86:6789/0 conn(0x7f2abc14faf0 legacy=0x7f2abc1e6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:05.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.620+0000 7f2ac2b75640 1 -- 172.21.15.84:0/2317046924 shutdown_connections 2024-03-29T10:38:05.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.620+0000 7f2ac2b75640 1 -- 172.21.15.84:0/2317046924 >> 172.21.15.84:0/2317046924 conn(0x7f2abc0b3c50 msgr2=0x7f2abc13f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:05.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.620+0000 7f2ac2b75640 1 -- 172.21.15.84:0/2317046924 shutdown_connections 2024-03-29T10:38:05.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.620+0000 7f2ac2b75640 1 -- 172.21.15.84:0/2317046924 wait complete. 2024-03-29T10:38:05.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:05.620+0000 7f2ac2b75640 1 librados: shutdown 2024-03-29T10:38:05.625 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:38:12.432 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:38:12.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.581+0000 7f4e1d80e640 10 monclient: get_monmap_and_config 2024-03-29T10:38:12.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.581+0000 7f4e1d80e640 10 monclient: build_initial_monmap 2024-03-29T10:38:12.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.581+0000 7f4e1d80e640 10 monclient: monmap: 2024-03-29T10:38:12.566 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:38:12.566 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:12.567 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:38:12.582919+0000 2024-03-29T10:38:12.567 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:38:12.582919+0000 2024-03-29T10:38:12.567 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:12.567 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:12.567 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:38:12.567 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:38:12.567 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:38:12.567 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:12.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.583+0000 7f4e1d80e640 1 Processor -- start 2024-03-29T10:38:12.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.584+0000 7f4e1d80e640 1 -- start start 2024-03-29T10:38:12.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.584+0000 7f4e1d80e640 10 monclient: init 2024-03-29T10:38:12.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.584+0000 7f4e1d80e640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:12.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.584+0000 7f4e1d80e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:38:12.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.584+0000 7f4e1d80e640 10 monclient(hunting): picked mon.noname-b con 0x7f4e181457f0 addr v1:172.21.15.84:6790/0 2024-03-29T10:38:12.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.585+0000 7f4e1d80e640 10 monclient(hunting): picked mon.noname-c con 0x7f4e1813f650 addr v1:172.21.15.86:6789/0 2024-03-29T10:38:12.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.585+0000 7f4e1d80e640 10 monclient(hunting): picked mon.noname-a con 0x7f4e180a9d70 addr v1:172.21.15.84:6789/0 2024-03-29T10:38:12.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.585+0000 7f4e177fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f4e180a9d70 0x7f4e1814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:60472/0 (socket says 172.21.15.84:60472) 2024-03-29T10:38:12.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.585+0000 7f4e177fe640 1 -- 172.21.15.84:0/1420101217 learned_addr learned my addr 172.21.15.84:0/1420101217 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:38:12.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.585+0000 7f4e1d80e640 1 -- 172.21.15.84:0/1420101217 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e180aaef0 con 0x7f4e180a9d70 2024-03-29T10:38:12.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.585+0000 7f4e16ffd640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f4e181457f0 0x7f4e1813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:41976/0 (socket says 172.21.15.84:41976) 2024-03-29T10:38:12.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.585+0000 7f4e1d80e640 1 -- 172.21.15.84:0/1420101217 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e18145bd0 con 0x7f4e1813f650 2024-03-29T10:38:12.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.585+0000 7f4e1d80e640 1 -- 172.21.15.84:0/1420101217 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e18145eb0 con 0x7f4e181457f0 2024-03-29T10:38:12.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.585+0000 7f4e15ffb640 1 -- 172.21.15.84:0/1420101217 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2300027056 0 0) 0x7f4e180aaef0 con 0x7f4e180a9d70 2024-03-29T10:38:12.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.585+0000 7f4e1d80e640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:12.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.585+0000 7f4e1d80e640 10 monclient(hunting): authenticate will time out at 994.848732s 2024-03-29T10:38:12.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.585+0000 7f4e15ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:38:12.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.585+0000 7f4e15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:12.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.585+0000 7f4e15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:12.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.585+0000 7f4e15ffb640 10 monclient(hunting): my global_id is 4350 2024-03-29T10:38:12.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.585+0000 7f4e15ffb640 1 -- 172.21.15.84:0/1420101217 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e00003520 con 0x7f4e180a9d70 2024-03-29T10:38:12.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.585+0000 7f4e15ffb640 1 -- 172.21.15.84:0/1420101217 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2724754 0 0) 0x7f4e18145eb0 con 0x7f4e181457f0 2024-03-29T10:38:12.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.585+0000 7f4e15ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:38:12.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.585+0000 7f4e15ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:12.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.585+0000 7f4e15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:12.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.585+0000 7f4e15ffb640 10 monclient(hunting): my global_id is 4376 2024-03-29T10:38:12.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.585+0000 7f4e15ffb640 1 -- 172.21.15.84:0/1420101217 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e180aaef0 con 0x7f4e181457f0 2024-03-29T10:38:12.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.585+0000 7f4e15ffb640 1 -- 172.21.15.84:0/1420101217 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2598115721 0 0) 0x7f4e00003520 con 0x7f4e180a9d70 2024-03-29T10:38:12.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.585+0000 7f4e15ffb640 1 -- 172.21.15.84:0/1420101217 >> v1:172.21.15.84:6790/0 conn(0x7f4e181457f0 legacy=0x7f4e1813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:12.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.585+0000 7f4e15ffb640 1 -- 172.21.15.84:0/1420101217 >> v1:172.21.15.86:6789/0 conn(0x7f4e1813f650 legacy=0x7f4e180b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:38:12.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.589+0000 7f4e15ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:38:12.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.589+0000 7f4e15ffb640 1 monclient: found mon.noname-a 2024-03-29T10:38:12.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.589+0000 7f4e15ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:12.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.589+0000 7f4e15ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T10:38:12.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.589+0000 7f4e15ffb640 1 -- 172.21.15.84:0/1420101217 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e18146340 con 0x7f4e180a9d70 2024-03-29T10:38:12.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.589+0000 7f4e15ffb640 10 monclient: _finish_auth 0 2024-03-29T10:38:12.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.589+0000 7f4e15ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:38:12.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.589+0000 7f4e15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:12.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.589+0000 7f4e15ffb640 1 -- 172.21.15.84:0/1420101217 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4e10001a10 con 0x7f4e180a9d70 2024-03-29T10:38:12.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.589+0000 7f4e15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:12.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.589+0000 7f4e15ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:38:12.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.589+0000 7f4e15ffb640 10 monclient: dump: 2024-03-29T10:38:12.579 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:12.579 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:12.579 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:12.579 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:12.579 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:12.579 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:12.579 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:12.579 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:12.579 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:12.579 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:12.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.589+0000 7f4e1d80e640 5 monclient: authenticate success, global_id 4350 2024-03-29T10:38:12.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.589+0000 7f4e1d80e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:12.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.589+0000 7f4e15ffb640 1 -- 172.21.15.84:0/1420101217 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e10001bf0 con 0x7f4e180a9d70 2024-03-29T10:38:12.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.589+0000 7f4e15ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:12.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.589+0000 7f4e15ffb640 1 -- 172.21.15.84:0/1420101217 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4e10002000 con 0x7f4e180a9d70 2024-03-29T10:38:12.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.589+0000 7f4e15ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:12.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.589+0000 7f4e15ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:12.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.589+0000 7f4e15ffb640 10 monclient: dump: 2024-03-29T10:38:12.580 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:12.580 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:12.580 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:12.580 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:12.580 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:12.581 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:12.581 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:12.581 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:12.581 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:12.581 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:12.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.589+0000 7f4e1d80e640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:12.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.589+0000 7f4e1d80e640 4 set_mon_vals no callback set 2024-03-29T10:38:12.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.590+0000 7f4e1d80e640 10 monclient: shutdown 2024-03-29T10:38:12.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.590+0000 7f4e1d80e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:12.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.590+0000 7f4e1d80e640 1 -- 172.21.15.84:0/1420101217 >> v1:172.21.15.84:6789/0 conn(0x7f4e180a9d70 legacy=0x7f4e1814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:12.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.590+0000 7f4e1d80e640 1 -- 172.21.15.84:0/1420101217 shutdown_connections 2024-03-29T10:38:12.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.590+0000 7f4e1d80e640 1 -- 172.21.15.84:0/1420101217 >> 172.21.15.84:0/1420101217 conn(0x7f4e18066a00 msgr2=0x7f4e18066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:12.582 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.590+0000 7f4e1d80e640 1 -- 172.21.15.84:0/1420101217 shutdown_connections 2024-03-29T10:38:12.582 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.591+0000 7f4e1d80e640 1 -- 172.21.15.84:0/1420101217 wait complete. 2024-03-29T10:38:12.582 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.592+0000 7f4e1d80e640 10 monclient: build_initial_monmap 2024-03-29T10:38:12.582 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.592+0000 7f4e1d80e640 10 monclient: monmap: 2024-03-29T10:38:12.582 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:38:12.582 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:12.582 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:38:12.583 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:38:12.583 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:12.583 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:12.583 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:38:12.583 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:38:12.583 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:38:12.583 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:12.583 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.592+0000 7f4e1d80e640 1 librados: starting msgr at 2024-03-29T10:38:12.583 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.592+0000 7f4e1d80e640 1 librados: starting objecter 2024-03-29T10:38:12.583 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.592+0000 7f4e1d80e640 1 Processor -- start 2024-03-29T10:38:12.583 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.592+0000 7f4e1d80e640 1 -- start start 2024-03-29T10:38:12.583 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.592+0000 7f4e1d80e640 1 librados: setting wanted keys 2024-03-29T10:38:12.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.592+0000 7f4e1d80e640 1 librados: calling monclient init 2024-03-29T10:38:12.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.592+0000 7f4e1d80e640 10 monclient: init 2024-03-29T10:38:12.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.593+0000 7f4e1d80e640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:12.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.593+0000 7f4e1d80e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:38:12.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.596+0000 7f4e1d80e640 10 monclient(hunting): picked mon.noname-b con 0x7f4e180a9d70 addr v1:172.21.15.86:6789/0 2024-03-29T10:38:12.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.596+0000 7f4e1d80e640 10 monclient(hunting): picked mon.noname-c con 0x7f4e1813f650 addr v1:172.21.15.84:6790/0 2024-03-29T10:38:12.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.596+0000 7f4e1d80e640 10 monclient(hunting): picked mon.noname-a con 0x7f4e181457f0 addr v1:172.21.15.84:6789/0 2024-03-29T10:38:12.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.596+0000 7f4e1d80e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e18141f30 con 0x7f4e181457f0 2024-03-29T10:38:12.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.596+0000 7f4e1d80e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e181421a0 con 0x7f4e180a9d70 2024-03-29T10:38:12.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.596+0000 7f4e1d80e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e18145170 con 0x7f4e1813f650 2024-03-29T10:38:12.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.596+0000 7f4e1d80e640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:12.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.596+0000 7f4e1d80e640 10 monclient(hunting): authenticate will time out at 994.859555s 2024-03-29T10:38:12.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.596+0000 7f4e177fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f4e181457f0 0x7f4e181e6bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:60758/0 (socket says 172.21.15.84:60758) 2024-03-29T10:38:12.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.596+0000 7f4e177fe640 1 -- 172.21.15.84:0/2907752635 learned_addr learned my addr 172.21.15.84:0/2907752635 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:38:12.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.596+0000 7f4e167fc640 1 --1- 172.21.15.84:0/2907752635 >> v1:172.21.15.84:6790/0 conn(0x7f4e1813f650 0x7f4e181e34a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:42356/0 (socket says 172.21.15.84:42356) 2024-03-29T10:38:12.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 10 client.?.objecter ms_handle_connect 0x7f4e181457f0 2024-03-29T10:38:12.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:12.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 10 client.?.objecter ms_handle_connect 0x7f4e1813f650 2024-03-29T10:38:12.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:12.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 1 -- 172.21.15.84:0/2907752635 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230014834 0 0) 0x7f4e18141f30 con 0x7f4e181457f0 2024-03-29T10:38:12.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 10 monclient(hunting): _negotiate 2024-03-29T10:38:12.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:12.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:12.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 10 monclient(hunting): my global_id is 4353 2024-03-29T10:38:12.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 1 -- 172.21.15.84:0/2907752635 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e04004eb0 con 0x7f4e181457f0 2024-03-29T10:38:12.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 1 -- 172.21.15.84:0/2907752635 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 98448864 0 0) 0x7f4e18145170 con 0x7f4e1813f650 2024-03-29T10:38:12.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 10 monclient(hunting): _negotiate 2024-03-29T10:38:12.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:12.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:12.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 10 monclient(hunting): my global_id is 4379 2024-03-29T10:38:12.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 1 -- 172.21.15.84:0/2907752635 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e18141f30 con 0x7f4e1813f650 2024-03-29T10:38:12.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 1 -- 172.21.15.84:0/2907752635 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2684111721 0 0) 0x7f4e18141f30 con 0x7f4e1813f650 2024-03-29T10:38:12.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 1 -- 172.21.15.84:0/2907752635 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e18145170 con 0x7f4e1813f650 2024-03-29T10:38:12.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 1 -- 172.21.15.84:0/2907752635 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4e0c0016e0 con 0x7f4e1813f650 2024-03-29T10:38:12.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:12.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:38:12.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 10 monclient(hunting): dump: 2024-03-29T10:38:12.594 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:12.594 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:12.594 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:12.595 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:12.595 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:12.595 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:12.595 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:12.595 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:12.595 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:12.595 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:12.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:12.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 1 -- 172.21.15.84:0/2907752635 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 220685495 0 0) 0x7f4e04004eb0 con 0x7f4e181457f0 2024-03-29T10:38:12.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 1 -- 172.21.15.84:0/2907752635 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e18141f30 con 0x7f4e181457f0 2024-03-29T10:38:12.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 1 -- 172.21.15.84:0/2907752635 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4e10001f00 con 0x7f4e181457f0 2024-03-29T10:38:12.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:12.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:12.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 10 monclient(hunting): dump: 2024-03-29T10:38:12.596 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:12.596 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:12.596 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:12.596 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:12.596 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:12.596 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:12.596 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:12.596 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:12.596 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:12.596 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:12.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 1 -- 172.21.15.84:0/2907752635 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 211429974 0 0) 0x7f4e18145170 con 0x7f4e1813f650 2024-03-29T10:38:12.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 1 -- 172.21.15.84:0/2907752635 >> v1:172.21.15.86:6789/0 conn(0x7f4e180a9d70 legacy=0x7f4e1813e010 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:38:12.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 1 -- 172.21.15.84:0/2907752635 >> v1:172.21.15.84:6789/0 conn(0x7f4e181457f0 legacy=0x7f4e181e6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:12.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 10 monclient: _finish_hunting 0 2024-03-29T10:38:12.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 1 monclient: found mon.c 2024-03-29T10:38:12.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:12.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:38:12.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 1 -- 172.21.15.84:0/2907752635 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e18135950 con 0x7f4e1813f650 2024-03-29T10:38:12.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 10 monclient: _finish_auth 0 2024-03-29T10:38:12.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 10 monclient: _check_auth_tickets 2024-03-29T10:38:12.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4dfffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:12.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4e1d80e640 5 monclient: authenticate success, global_id 4379 2024-03-29T10:38:12.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4e1d80e640 10 monclient: _renew_subs 2024-03-29T10:38:12.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4e1d80e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:38:12.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4e1d80e640 1 -- 172.21.15.84:0/2907752635 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e18140f80 con 0x7f4e1813f650 2024-03-29T10:38:12.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4e1d80e640 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:12.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4e1d80e640 10 monclient: _renew_subs 2024-03-29T10:38:12.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4e1d80e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:38:12.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4e1d80e640 1 -- 172.21.15.84:0/2907752635 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e18140af0 con 0x7f4e1813f650 2024-03-29T10:38:12.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.597+0000 7f4e1d80e640 1 librados: init done 2024-03-29T10:38:12.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.598+0000 7f4dfffff640 1 -- 172.21.15.84:0/2907752635 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e0c001840 con 0x7f4e1813f650 2024-03-29T10:38:12.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.598+0000 7f4dfffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:12.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.598+0000 7f4dfffff640 1 -- 172.21.15.84:0/2907752635 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4e0c004840 con 0x7f4e1813f650 2024-03-29T10:38:12.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.598+0000 7f4dfffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:12.599 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.598+0000 7f4dfffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:12.599 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.598+0000 7f4dfffff640 10 monclient: dump: 2024-03-29T10:38:12.599 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:12.599 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:12.599 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:12.599 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:12.599 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:12.599 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:12.599 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:12.599 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:12.599 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:12.599 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:12.599 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.599+0000 7f4e157fa640 4 set_mon_vals no callback set 2024-03-29T10:38:12.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.599+0000 7f4dfffff640 1 -- 172.21.15.84:0/2907752635 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f4e0c01a810 con 0x7f4e1813f650 2024-03-29T10:38:12.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.600+0000 7f4e1d80e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:12.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.600+0000 7f4e1d80e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:12.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.600+0000 7f4e1d80e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:38:12.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.600+0000 7f4e1d80e640 1 -- 172.21.15.84:0/2907752635 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df8005080 con 0x7f4e1813f650 2024-03-29T10:38:12.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.601+0000 7f4dfffff640 1 -- 172.21.15.84:0/2907752635 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f4e0c084840 con 0x7f4e1813f650 2024-03-29T10:38:12.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.601+0000 7f4dfffff640 10 client.4379.objecter ms_dispatch 0x7f4e18000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:38:12.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.601+0000 7f4e16ffd640 10 monclient: get_auth_request con 0x7f4e0406df40 auth_method 0 2024-03-29T10:38:12.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.601+0000 7f4dfffff640 3 client.4379.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:38:12.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.601+0000 7f4dfffff640 3 client.4379.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:38:12.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.601+0000 7f4dfffff640 20 client.4379.objecter dump_active .. 0 homeless 2024-03-29T10:38:12.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.601+0000 7f4dfffff640 10 client.4379.objecter ms_handle_connect 0x7f4e0406df40 2024-03-29T10:38:12.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.607+0000 7f4dfffff640 1 -- 172.21.15.84:0/2907752635 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4e0c050120 con 0x7f4e1813f650 2024-03-29T10:38:12.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.607+0000 7f4dfffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:12.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.607+0000 7f4dfffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:12.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.945+0000 7f4e1d80e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:12.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.945+0000 7f4e1d80e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:12.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.945+0000 7f4e1d80e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:38:12.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.945+0000 7f4e1d80e640 1 -- 172.21.15.84:0/2907752635 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4df8005330 con 0x7f4e1813f650 2024-03-29T10:38:12.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.947+0000 7f4dfffff640 1 -- 172.21.15.84:0/2907752635 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f4e0c052050 con 0x7f4e1813f650 2024-03-29T10:38:12.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.947+0000 7f4dfffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:12.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.947+0000 7f4dfffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:12.931 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:38:12.932 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:38:12.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.950+0000 7f4e1d80e640 10 librados: watch_flush enter 2024-03-29T10:38:12.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.951+0000 7f4e1d80e640 10 librados: watch_flush exit 2024-03-29T10:38:12.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.951+0000 7f4e1d80e640 20 client.4379.objecter shutdown clearing up homeless session... 2024-03-29T10:38:12.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.951+0000 7f4e1d80e640 10 client.4379.objecter successfully canceled tick 2024-03-29T10:38:12.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.951+0000 7f4e1d80e640 1 -- 172.21.15.84:0/2907752635 >> v1:172.21.15.86:6816/3337225337 conn(0x7f4e0406df40 legacy=0x7f4e04070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:12.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.951+0000 7f4e1d80e640 10 monclient: shutdown 2024-03-29T10:38:12.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.951+0000 7f4e1d80e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:12.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.951+0000 7f4e1d80e640 1 -- 172.21.15.84:0/2907752635 >> v1:172.21.15.84:6790/0 conn(0x7f4e1813f650 legacy=0x7f4e181e34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:12.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.951+0000 7f4e1d80e640 1 -- 172.21.15.84:0/2907752635 shutdown_connections 2024-03-29T10:38:12.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.951+0000 7f4e1d80e640 1 -- 172.21.15.84:0/2907752635 >> 172.21.15.84:0/2907752635 conn(0x7f4e18066a00 msgr2=0x7f4e180b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:12.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.952+0000 7f4e1d80e640 1 -- 172.21.15.84:0/2907752635 shutdown_connections 2024-03-29T10:38:12.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.952+0000 7f4e1d80e640 1 -- 172.21.15.84:0/2907752635 wait complete. 2024-03-29T10:38:12.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:12.952+0000 7f4e1d80e640 1 librados: shutdown 2024-03-29T10:38:12.953 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:38:19.760 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:38:19.902 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.918+0000 7f535e87d640 10 monclient: get_monmap_and_config 2024-03-29T10:38:19.902 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.918+0000 7f535e87d640 10 monclient: build_initial_monmap 2024-03-29T10:38:19.902 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.918+0000 7f535e87d640 10 monclient: monmap: 2024-03-29T10:38:19.902 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:38:19.903 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:19.903 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:38:19.919150+0000 2024-03-29T10:38:19.903 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:38:19.919150+0000 2024-03-29T10:38:19.903 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:19.903 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:19.903 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:38:19.903 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:38:19.903 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:38:19.903 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:19.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.919+0000 7f535e87d640 1 Processor -- start 2024-03-29T10:38:19.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.919+0000 7f535e87d640 1 -- start start 2024-03-29T10:38:19.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.919+0000 7f535e87d640 10 monclient: init 2024-03-29T10:38:19.905 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.920+0000 7f535e87d640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:19.905 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.920+0000 7f535e87d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:38:19.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.921+0000 7f535e87d640 10 monclient(hunting): picked mon.noname-b con 0x7f53581422b0 addr v1:172.21.15.84:6790/0 2024-03-29T10:38:19.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.921+0000 7f535e87d640 10 monclient(hunting): picked mon.noname-a con 0x7f5358142690 addr v1:172.21.15.84:6789/0 2024-03-29T10:38:19.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.921+0000 7f535e87d640 10 monclient(hunting): picked mon.noname-c con 0x7f53580b17d0 addr v1:172.21.15.86:6789/0 2024-03-29T10:38:19.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.921+0000 7f535e87d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5358142c80 con 0x7f5358142690 2024-03-29T10:38:19.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.921+0000 7f5357fff640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f53581422b0 0x7f535813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:33174/0 (socket says 172.21.15.84:33174) 2024-03-29T10:38:19.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.921+0000 7f5357fff640 1 -- 172.21.15.84:0/3795630982 learned_addr learned my addr 172.21.15.84:0/3795630982 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:38:19.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.921+0000 7f535e87d640 1 -- 172.21.15.84:0/3795630982 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53580aae40 con 0x7f53580b17d0 2024-03-29T10:38:19.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.921+0000 7f535e87d640 1 -- 172.21.15.84:0/3795630982 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535813fbd0 con 0x7f53581422b0 2024-03-29T10:38:19.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.921+0000 7f535e87d640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:19.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.921+0000 7f535e87d640 10 monclient(hunting): authenticate will time out at 1002.184761s 2024-03-29T10:38:19.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.922+0000 7f5356ffd640 1 -- 172.21.15.84:0/3795630982 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2083388942 0 0) 0x7f535813fbd0 con 0x7f53581422b0 2024-03-29T10:38:19.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.922+0000 7f5356ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:38:19.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.922+0000 7f5356ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:19.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.922+0000 7f5356ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:19.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.922+0000 7f5356ffd640 10 monclient(hunting): my global_id is 4382 2024-03-29T10:38:19.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.922+0000 7f5356ffd640 1 -- 172.21.15.84:0/3795630982 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5340003520 con 0x7f53581422b0 2024-03-29T10:38:19.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.922+0000 7f5356ffd640 1 -- 172.21.15.84:0/3795630982 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550573851 0 0) 0x7f5358142c80 con 0x7f5358142690 2024-03-29T10:38:19.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.922+0000 7f5356ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:38:19.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.922+0000 7f5356ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:19.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.922+0000 7f5356ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:19.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.922+0000 7f5356ffd640 10 monclient(hunting): my global_id is 4356 2024-03-29T10:38:19.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.922+0000 7f5356ffd640 1 -- 172.21.15.84:0/3795630982 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f535813fbd0 con 0x7f5358142690 2024-03-29T10:38:19.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.922+0000 7f5356ffd640 1 -- 172.21.15.84:0/3795630982 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3441172848 0 0) 0x7f5340003520 con 0x7f53581422b0 2024-03-29T10:38:19.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.922+0000 7f5356ffd640 1 -- 172.21.15.84:0/3795630982 >> v1:172.21.15.86:6789/0 conn(0x7f53580b17d0 legacy=0x7f53580b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:19.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.922+0000 7f5356ffd640 1 -- 172.21.15.84:0/3795630982 >> v1:172.21.15.84:6789/0 conn(0x7f5358142690 legacy=0x7f53580b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:19.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.922+0000 7f5356ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:38:19.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.922+0000 7f5356ffd640 1 monclient: found mon.noname-b 2024-03-29T10:38:19.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.922+0000 7f5356ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:19.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.922+0000 7f5356ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:38:19.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.923+0000 7f5356ffd640 1 -- 172.21.15.84:0/3795630982 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5358151940 con 0x7f53581422b0 2024-03-29T10:38:19.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.923+0000 7f5356ffd640 10 monclient: _finish_auth 0 2024-03-29T10:38:19.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.923+0000 7f5356ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:38:19.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.923+0000 7f5356ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:19.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.923+0000 7f5356ffd640 1 -- 172.21.15.84:0/3795630982 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f534c0015c0 con 0x7f53581422b0 2024-03-29T10:38:19.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.923+0000 7f535e87d640 5 monclient: authenticate success, global_id 4382 2024-03-29T10:38:19.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.923+0000 7f535e87d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:19.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.923+0000 7f5356ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:19.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.923+0000 7f5356ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:38:19.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.923+0000 7f5356ffd640 10 monclient: dump: 2024-03-29T10:38:19.929 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:19.929 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:19.930 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:19.930 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:19.930 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:19.930 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:19.930 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:19.930 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:19.930 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:19.930 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:19.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.923+0000 7f535e87d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:19.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.923+0000 7f5356ffd640 1 -- 172.21.15.84:0/3795630982 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f534c001720 con 0x7f53581422b0 2024-03-29T10:38:19.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.923+0000 7f5356ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:19.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.923+0000 7f5356ffd640 1 -- 172.21.15.84:0/3795630982 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f534c001ab0 con 0x7f53581422b0 2024-03-29T10:38:19.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.923+0000 7f5356ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:19.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.923+0000 7f5356ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:19.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.923+0000 7f5356ffd640 10 monclient: dump: 2024-03-29T10:38:19.932 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:19.932 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:19.932 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:19.932 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:19.932 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:19.932 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:19.932 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:19.932 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:19.932 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:19.932 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:19.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.923+0000 7f535e87d640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:19.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.923+0000 7f535e87d640 4 set_mon_vals no callback set 2024-03-29T10:38:19.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.924+0000 7f535e87d640 10 monclient: shutdown 2024-03-29T10:38:19.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.924+0000 7f535e87d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:19.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.924+0000 7f535e87d640 1 -- 172.21.15.84:0/3795630982 >> v1:172.21.15.84:6790/0 conn(0x7f53581422b0 legacy=0x7f535813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:19.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.924+0000 7f535e87d640 1 -- 172.21.15.84:0/3795630982 shutdown_connections 2024-03-29T10:38:19.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.924+0000 7f535e87d640 1 -- 172.21.15.84:0/3795630982 >> 172.21.15.84:0/3795630982 conn(0x7f5358066a00 msgr2=0x7f5358066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:19.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.925+0000 7f535e87d640 1 -- 172.21.15.84:0/3795630982 shutdown_connections 2024-03-29T10:38:19.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.925+0000 7f535e87d640 1 -- 172.21.15.84:0/3795630982 wait complete. 2024-03-29T10:38:19.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.926+0000 7f535e87d640 10 monclient: build_initial_monmap 2024-03-29T10:38:19.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.926+0000 7f535e87d640 10 monclient: monmap: 2024-03-29T10:38:19.934 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:38:19.934 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:19.934 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:38:19.934 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:38:19.934 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:19.934 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:19.934 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:38:19.934 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:38:19.934 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:38:19.934 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:19.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.927+0000 7f535e87d640 1 librados: starting msgr at 2024-03-29T10:38:19.934 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.927+0000 7f535e87d640 1 librados: starting objecter 2024-03-29T10:38:19.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.927+0000 7f535e87d640 1 Processor -- start 2024-03-29T10:38:19.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.927+0000 7f535e87d640 1 -- start start 2024-03-29T10:38:19.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.927+0000 7f535e87d640 1 librados: setting wanted keys 2024-03-29T10:38:19.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.927+0000 7f535e87d640 1 librados: calling monclient init 2024-03-29T10:38:19.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.927+0000 7f535e87d640 10 monclient: init 2024-03-29T10:38:19.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.928+0000 7f535e87d640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:19.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.928+0000 7f535e87d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:38:19.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.928+0000 7f535e87d640 10 monclient(hunting): picked mon.noname-c con 0x7f53580b17d0 addr v1:172.21.15.84:6790/0 2024-03-29T10:38:19.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.928+0000 7f535e87d640 10 monclient(hunting): picked mon.noname-b con 0x7f53581422b0 addr v1:172.21.15.86:6789/0 2024-03-29T10:38:19.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.928+0000 7f5357fff640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f53580b17d0 0x7f535812ecb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:33392/0 (socket says 172.21.15.84:33392) 2024-03-29T10:38:19.935 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.928+0000 7f535e87d640 10 monclient(hunting): picked mon.noname-a con 0x7f5358142690 addr v1:172.21.15.84:6789/0 2024-03-29T10:38:19.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.928+0000 7f535e87d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53580ac730 con 0x7f5358142690 2024-03-29T10:38:19.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.928+0000 7f535ce1a640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f5358142690 0x7f53581e2d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:43526/0 (socket says 172.21.15.84:43526) 2024-03-29T10:38:19.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.928+0000 7f535ce1a640 1 -- 172.21.15.84:0/1618087128 learned_addr learned my addr 172.21.15.84:0/1618087128 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:38:19.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.928+0000 7f535e87d640 1 -- 172.21.15.84:0/1618087128 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53580ac9a0 con 0x7f53581422b0 2024-03-29T10:38:19.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.928+0000 7f535e87d640 1 -- 172.21.15.84:0/1618087128 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53580acc80 con 0x7f53580b17d0 2024-03-29T10:38:19.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.928+0000 7f535e87d640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:19.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.928+0000 7f535e87d640 10 monclient(hunting): authenticate will time out at 1002.191927s 2024-03-29T10:38:19.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.928+0000 7f5354ff9640 10 client.?.objecter ms_handle_connect 0x7f5358142690 2024-03-29T10:38:19.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.928+0000 7f5354ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:19.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.928+0000 7f5354ff9640 10 client.?.objecter ms_handle_connect 0x7f53580b17d0 2024-03-29T10:38:19.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.928+0000 7f5354ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:19.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.929+0000 7f5354ff9640 10 client.?.objecter ms_handle_connect 0x7f53581422b0 2024-03-29T10:38:19.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.929+0000 7f5354ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:19.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.929+0000 7f5354ff9640 1 -- 172.21.15.84:0/1618087128 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4209965966 0 0) 0x7f53580ac730 con 0x7f5358142690 2024-03-29T10:38:19.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.929+0000 7f5354ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:38:19.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.929+0000 7f5354ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:19.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.929+0000 7f5354ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:19.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.929+0000 7f5354ff9640 10 monclient(hunting): my global_id is 4359 2024-03-29T10:38:19.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.929+0000 7f5354ff9640 1 -- 172.21.15.84:0/1618087128 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5344004eb0 con 0x7f5358142690 2024-03-29T10:38:19.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.929+0000 7f5354ff9640 1 -- 172.21.15.84:0/1618087128 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3284317565 0 0) 0x7f53580acc80 con 0x7f53580b17d0 2024-03-29T10:38:19.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.929+0000 7f5354ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:38:19.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.929+0000 7f5354ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:19.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.929+0000 7f5354ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:19.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.929+0000 7f5354ff9640 10 monclient(hunting): my global_id is 4385 2024-03-29T10:38:19.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.929+0000 7f5354ff9640 1 -- 172.21.15.84:0/1618087128 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53580ac730 con 0x7f53580b17d0 2024-03-29T10:38:19.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.929+0000 7f5354ff9640 1 -- 172.21.15.84:0/1618087128 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430505775 0 0) 0x7f53580ac9a0 con 0x7f53581422b0 2024-03-29T10:38:19.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.929+0000 7f5354ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:38:19.938 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.929+0000 7f5354ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:19.938 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.929+0000 7f5354ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:19.938 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.929+0000 7f5354ff9640 10 monclient(hunting): my global_id is 4345 2024-03-29T10:38:19.938 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.929+0000 7f5354ff9640 1 -- 172.21.15.84:0/1618087128 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53580acc80 con 0x7f53581422b0 2024-03-29T10:38:19.938 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.930+0000 7f5354ff9640 1 -- 172.21.15.84:0/1618087128 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4200928090 0 0) 0x7f53580ac730 con 0x7f53580b17d0 2024-03-29T10:38:19.938 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.930+0000 7f5354ff9640 1 -- 172.21.15.84:0/1618087128 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53580ac9a0 con 0x7f53580b17d0 2024-03-29T10:38:19.938 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.930+0000 7f5354ff9640 1 -- 172.21.15.84:0/1618087128 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f534c001a30 con 0x7f53580b17d0 2024-03-29T10:38:19.938 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.930+0000 7f5354ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:19.938 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.930+0000 7f5354ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:38:19.938 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.930+0000 7f5354ff9640 10 monclient(hunting): dump: 2024-03-29T10:38:19.938 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:19.938 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:19.938 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:19.938 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:19.939 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:19.939 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:19.939 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:19.939 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:19.939 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:19.939 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:19.939 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.930+0000 7f5354ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:19.939 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.930+0000 7f5354ff9640 1 -- 172.21.15.84:0/1618087128 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3782574522 0 0) 0x7f53580acc80 con 0x7f53581422b0 2024-03-29T10:38:19.939 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.930+0000 7f5354ff9640 1 -- 172.21.15.84:0/1618087128 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53580ac730 con 0x7f53581422b0 2024-03-29T10:38:19.939 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.930+0000 7f5354ff9640 1 -- 172.21.15.84:0/1618087128 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 908611192 0 0) 0x7f5344004eb0 con 0x7f5358142690 2024-03-29T10:38:19.939 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.930+0000 7f5354ff9640 1 -- 172.21.15.84:0/1618087128 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53580acc80 con 0x7f5358142690 2024-03-29T10:38:19.939 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.930+0000 7f5354ff9640 1 -- 172.21.15.84:0/1618087128 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f53480017d0 con 0x7f53581422b0 2024-03-29T10:38:19.939 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.930+0000 7f5354ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:19.939 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.930+0000 7f5354ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:19.939 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.930+0000 7f5354ff9640 10 monclient(hunting): dump: 2024-03-29T10:38:19.940 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:19.940 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:19.940 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:19.940 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:19.940 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:19.940 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:19.940 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:19.940 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:19.940 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:19.940 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:19.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.930+0000 7f5354ff9640 1 -- 172.21.15.84:0/1618087128 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f5350001c50 con 0x7f5358142690 2024-03-29T10:38:19.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.930+0000 7f5354ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:19.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.930+0000 7f5354ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:19.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.930+0000 7f5354ff9640 10 monclient(hunting): dump: 2024-03-29T10:38:19.941 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:19.941 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:19.941 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:19.941 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:19.941 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:19.941 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:19.941 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:19.941 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:19.941 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:19.941 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:19.941 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.930+0000 7f5354ff9640 1 -- 172.21.15.84:0/1618087128 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1885519791 0 0) 0x7f53580ac9a0 con 0x7f53580b17d0 2024-03-29T10:38:19.941 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.931+0000 7f5354ff9640 1 -- 172.21.15.84:0/1618087128 >> v1:172.21.15.86:6789/0 conn(0x7f53581422b0 legacy=0x7f535812fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:19.941 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.931+0000 7f5354ff9640 1 -- 172.21.15.84:0/1618087128 >> v1:172.21.15.84:6789/0 conn(0x7f5358142690 legacy=0x7f53581e2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:19.941 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.931+0000 7f5354ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:38:19.941 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.931+0000 7f5354ff9640 1 monclient: found mon.c 2024-03-29T10:38:19.942 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.931+0000 7f5354ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:19.942 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.931+0000 7f5354ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:38:19.942 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.931+0000 7f5354ff9640 1 -- 172.21.15.84:0/1618087128 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53580acf60 con 0x7f53580b17d0 2024-03-29T10:38:19.942 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.931+0000 7f5354ff9640 10 monclient: _finish_auth 0 2024-03-29T10:38:19.942 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.931+0000 7f5354ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:38:19.942 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.931+0000 7f5354ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:19.942 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.931+0000 7f535e87d640 5 monclient: authenticate success, global_id 4385 2024-03-29T10:38:19.942 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.931+0000 7f535e87d640 10 monclient: _renew_subs 2024-03-29T10:38:19.942 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.931+0000 7f535e87d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:38:19.942 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.931+0000 7f535e87d640 1 -- 172.21.15.84:0/1618087128 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5358132740 con 0x7f53580b17d0 2024-03-29T10:38:19.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.931+0000 7f535e87d640 10 client.4385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:19.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.931+0000 7f535e87d640 10 monclient: _renew_subs 2024-03-29T10:38:19.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.931+0000 7f5354ff9640 1 -- 172.21.15.84:0/1618087128 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f534c004a90 con 0x7f53580b17d0 2024-03-29T10:38:19.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.931+0000 7f535e87d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:38:19.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.931+0000 7f535e87d640 1 -- 172.21.15.84:0/1618087128 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f535812f3c0 con 0x7f53580b17d0 2024-03-29T10:38:19.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.931+0000 7f5354ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:19.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.931+0000 7f5354ff9640 1 -- 172.21.15.84:0/1618087128 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f534c004da0 con 0x7f53580b17d0 2024-03-29T10:38:19.943 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.931+0000 7f5354ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:19.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.931+0000 7f5354ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:19.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.931+0000 7f5354ff9640 10 monclient: dump: 2024-03-29T10:38:19.944 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:19.944 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:19.944 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:19.944 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:19.944 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:19.944 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:19.945 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:19.945 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:19.945 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:19.945 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:19.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.931+0000 7f53567fc640 4 set_mon_vals no callback set 2024-03-29T10:38:19.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.931+0000 7f535e87d640 1 librados: init done 2024-03-29T10:38:19.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.933+0000 7f5354ff9640 1 -- 172.21.15.84:0/1618087128 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f534c01ad70 con 0x7f53580b17d0 2024-03-29T10:38:19.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.933+0000 7f535e87d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:19.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.933+0000 7f535e87d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:19.946 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.933+0000 7f535e87d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:38:19.946 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.933+0000 7f535e87d640 1 -- 172.21.15.84:0/1618087128 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5358132540 con 0x7f53580b17d0 2024-03-29T10:38:19.946 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.934+0000 7f5354ff9640 1 -- 172.21.15.84:0/1618087128 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f534c084a40 con 0x7f53580b17d0 2024-03-29T10:38:19.946 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.934+0000 7f5354ff9640 10 client.4385.objecter ms_dispatch 0x7f5358000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:38:19.946 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.934+0000 7f5354ff9640 3 client.4385.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:38:19.946 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.934+0000 7f5354ff9640 3 client.4385.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:38:19.946 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.934+0000 7f53577fe640 10 monclient: get_auth_request con 0x7f534406e220 auth_method 0 2024-03-29T10:38:19.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.934+0000 7f5354ff9640 20 client.4385.objecter dump_active .. 0 homeless 2024-03-29T10:38:19.947 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.934+0000 7f5354ff9640 10 client.4385.objecter ms_handle_connect 0x7f534406e220 2024-03-29T10:38:19.948 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.941+0000 7f5354ff9640 1 -- 172.21.15.84:0/1618087128 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f534c050380 con 0x7f53580b17d0 2024-03-29T10:38:19.948 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.941+0000 7f5354ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:19.948 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:19.941+0000 7f5354ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:20.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:20.293+0000 7f535e87d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:20.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:20.293+0000 7f535e87d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:20.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:20.293+0000 7f535e87d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:38:20.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:20.293+0000 7f535e87d640 1 -- 172.21.15.84:0/1618087128 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53581328d0 con 0x7f53580b17d0 2024-03-29T10:38:20.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:20.294+0000 7f5354ff9640 1 -- 172.21.15.84:0/1618087128 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f534c0840f0 con 0x7f53580b17d0 2024-03-29T10:38:20.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:20.294+0000 7f5354ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:20.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:20.294+0000 7f5354ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:20.279 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:38:20.279 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:38:20.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:20.298+0000 7f535e87d640 10 librados: watch_flush enter 2024-03-29T10:38:20.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:20.298+0000 7f535e87d640 10 librados: watch_flush exit 2024-03-29T10:38:20.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:20.298+0000 7f535e87d640 20 client.4385.objecter shutdown clearing up homeless session... 2024-03-29T10:38:20.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:20.298+0000 7f535e87d640 10 client.4385.objecter successfully canceled tick 2024-03-29T10:38:20.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:20.298+0000 7f535e87d640 1 -- 172.21.15.84:0/1618087128 >> v1:172.21.15.86:6816/3337225337 conn(0x7f534406e220 legacy=0x7f53440706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:20.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:20.298+0000 7f535e87d640 10 monclient: shutdown 2024-03-29T10:38:20.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:20.298+0000 7f535e87d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:20.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:20.298+0000 7f535e87d640 1 -- 172.21.15.84:0/1618087128 >> v1:172.21.15.84:6790/0 conn(0x7f53580b17d0 legacy=0x7f535812ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:20.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:20.298+0000 7f535e87d640 1 -- 172.21.15.84:0/1618087128 shutdown_connections 2024-03-29T10:38:20.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:20.298+0000 7f535e87d640 1 -- 172.21.15.84:0/1618087128 >> 172.21.15.84:0/1618087128 conn(0x7f5358066a00 msgr2=0x7f53580b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:20.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:20.298+0000 7f535e87d640 1 -- 172.21.15.84:0/1618087128 shutdown_connections 2024-03-29T10:38:20.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:20.298+0000 7f535e87d640 1 -- 172.21.15.84:0/1618087128 wait complete. 2024-03-29T10:38:20.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:20.298+0000 7f535e87d640 1 librados: shutdown 2024-03-29T10:38:20.297 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:38:27.104 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:38:27.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.273+0000 7fb02000e640 10 monclient: get_monmap_and_config 2024-03-29T10:38:27.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.273+0000 7fb02000e640 10 monclient: build_initial_monmap 2024-03-29T10:38:27.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.273+0000 7fb02000e640 10 monclient: monmap: 2024-03-29T10:38:27.258 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:38:27.258 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:27.259 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:38:27.274855+0000 2024-03-29T10:38:27.259 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:38:27.274855+0000 2024-03-29T10:38:27.259 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:27.259 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:27.259 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:38:27.259 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:38:27.259 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:38:27.259 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:27.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.277+0000 7fb02000e640 1 Processor -- start 2024-03-29T10:38:27.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.277+0000 7fb02000e640 1 -- start start 2024-03-29T10:38:27.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.277+0000 7fb02000e640 10 monclient: init 2024-03-29T10:38:27.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.278+0000 7fb02000e640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:27.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.278+0000 7fb02000e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:38:27.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.278+0000 7fb02000e640 10 monclient(hunting): picked mon.noname-b con 0x7fb0180b4330 addr v1:172.21.15.84:6790/0 2024-03-29T10:38:27.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.279+0000 7fb02000e640 10 monclient(hunting): picked mon.noname-c con 0x7fb0181470b0 addr v1:172.21.15.86:6789/0 2024-03-29T10:38:27.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.279+0000 7fb02000e640 10 monclient(hunting): picked mon.noname-a con 0x7fb0181439b0 addr v1:172.21.15.84:6789/0 2024-03-29T10:38:27.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.279+0000 7fb02000e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb01806f2a0 con 0x7fb0181439b0 2024-03-29T10:38:27.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.279+0000 7fb02000e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb018146b90 con 0x7fb0181470b0 2024-03-29T10:38:27.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.279+0000 7fb02000e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0180b2ca0 con 0x7fb0180b4330 2024-03-29T10:38:27.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.279+0000 7fb02000e640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:27.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.279+0000 7fb02000e640 10 monclient(hunting): authenticate will time out at 1009.542544s 2024-03-29T10:38:27.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.280+0000 7fb01d5a9640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fb0181470b0 0x7fb0180b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:48588/0 (socket says 172.21.15.84:48588) 2024-03-29T10:38:27.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.280+0000 7fb01d5a9640 1 -- 172.21.15.84:0/309232926 learned_addr learned my addr 172.21.15.84:0/309232926 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:38:27.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.282+0000 7fb01cda8640 1 -- 172.21.15.84:0/309232926 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2094739748 0 0) 0x7fb018146b90 con 0x7fb0181470b0 2024-03-29T10:38:27.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.282+0000 7fb01cda8640 10 monclient(hunting): _negotiate 2024-03-29T10:38:27.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.282+0000 7fb01cda8640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:27.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.282+0000 7fb01cda8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:27.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.282+0000 7fb01cda8640 10 monclient(hunting): my global_id is 4348 2024-03-29T10:38:27.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.282+0000 7fb01cda8640 1 -- 172.21.15.84:0/309232926 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb004003520 con 0x7fb0181470b0 2024-03-29T10:38:27.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.284+0000 7fb01cda8640 1 -- 172.21.15.84:0/309232926 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2808538205 0 0) 0x7fb01806f2a0 con 0x7fb0181439b0 2024-03-29T10:38:27.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.284+0000 7fb01cda8640 10 monclient(hunting): _negotiate 2024-03-29T10:38:27.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.284+0000 7fb01cda8640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:27.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.284+0000 7fb01cda8640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:27.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.284+0000 7fb01cda8640 10 monclient(hunting): my global_id is 4362 2024-03-29T10:38:27.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.284+0000 7fb01cda8640 1 -- 172.21.15.84:0/309232926 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb018146b90 con 0x7fb0181439b0 2024-03-29T10:38:27.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.284+0000 7fb01cda8640 1 -- 172.21.15.84:0/309232926 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3906971208 0 0) 0x7fb018146b90 con 0x7fb0181439b0 2024-03-29T10:38:27.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.284+0000 7fb01cda8640 1 -- 172.21.15.84:0/309232926 >> v1:172.21.15.84:6790/0 conn(0x7fb0180b4330 legacy=0x7fb0180b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:38:27.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.284+0000 7fb01cda8640 1 -- 172.21.15.84:0/309232926 >> v1:172.21.15.86:6789/0 conn(0x7fb0181470b0 legacy=0x7fb0180b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:27.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.284+0000 7fb01cda8640 10 monclient: _finish_hunting 0 2024-03-29T10:38:27.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.284+0000 7fb01cda8640 1 monclient: found mon.noname-a 2024-03-29T10:38:27.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.284+0000 7fb01cda8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:27.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.284+0000 7fb01cda8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T10:38:27.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.284+0000 7fb01cda8640 1 -- 172.21.15.84:0/309232926 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb01806e0c0 con 0x7fb0181439b0 2024-03-29T10:38:27.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.284+0000 7fb01cda8640 10 monclient: _finish_auth 0 2024-03-29T10:38:27.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.284+0000 7fb01cda8640 10 monclient: _check_auth_tickets 2024-03-29T10:38:27.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.284+0000 7fb01cda8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:27.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.284+0000 7fb01cda8640 1 -- 172.21.15.84:0/309232926 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fb0100019f0 con 0x7fb0181439b0 2024-03-29T10:38:27.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.284+0000 7fb01cda8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:27.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.284+0000 7fb01cda8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:38:27.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.284+0000 7fb01cda8640 10 monclient: dump: 2024-03-29T10:38:27.275 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:27.275 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:27.275 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:27.275 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:27.275 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:27.275 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:27.275 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:27.275 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:27.275 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:27.276 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:27.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.285+0000 7fb02000e640 5 monclient: authenticate success, global_id 4362 2024-03-29T10:38:27.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.285+0000 7fb02000e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:27.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.285+0000 7fb01cda8640 1 -- 172.21.15.84:0/309232926 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb010001b50 con 0x7fb0181439b0 2024-03-29T10:38:27.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.285+0000 7fb01cda8640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:27.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.285+0000 7fb01cda8640 1 -- 172.21.15.84:0/309232926 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fb010001f60 con 0x7fb0181439b0 2024-03-29T10:38:27.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.285+0000 7fb01cda8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:27.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.285+0000 7fb01cda8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:27.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.285+0000 7fb01cda8640 10 monclient: dump: 2024-03-29T10:38:27.276 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:27.276 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:27.276 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:27.276 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:27.276 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:27.276 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:27.277 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:27.277 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:27.277 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:27.277 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:27.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.285+0000 7fb02000e640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:27.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.285+0000 7fb02000e640 4 set_mon_vals no callback set 2024-03-29T10:38:27.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.286+0000 7fb02000e640 10 monclient: shutdown 2024-03-29T10:38:27.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.286+0000 7fb02000e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:27.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.286+0000 7fb02000e640 1 -- 172.21.15.84:0/309232926 >> v1:172.21.15.84:6789/0 conn(0x7fb0181439b0 legacy=0x7fb018151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:27.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.286+0000 7fb02000e640 1 -- 172.21.15.84:0/309232926 shutdown_connections 2024-03-29T10:38:27.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.286+0000 7fb02000e640 1 -- 172.21.15.84:0/309232926 >> 172.21.15.84:0/309232926 conn(0x7fb018066a00 msgr2=0x7fb018066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:27.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.287+0000 7fb02000e640 1 -- 172.21.15.84:0/309232926 shutdown_connections 2024-03-29T10:38:27.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.287+0000 7fb02000e640 1 -- 172.21.15.84:0/309232926 wait complete. 2024-03-29T10:38:27.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.288+0000 7fb02000e640 10 monclient: build_initial_monmap 2024-03-29T10:38:27.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.288+0000 7fb02000e640 10 monclient: monmap: 2024-03-29T10:38:27.278 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:38:27.278 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:27.278 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:38:27.278 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:38:27.278 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:27.278 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:27.278 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:38:27.278 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:38:27.278 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:38:27.278 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:27.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.288+0000 7fb02000e640 1 librados: starting msgr at 2024-03-29T10:38:27.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.288+0000 7fb02000e640 1 librados: starting objecter 2024-03-29T10:38:27.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.289+0000 7fb02000e640 1 Processor -- start 2024-03-29T10:38:27.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.289+0000 7fb02000e640 1 -- start start 2024-03-29T10:38:27.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.289+0000 7fb02000e640 1 librados: setting wanted keys 2024-03-29T10:38:27.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.289+0000 7fb02000e640 1 librados: calling monclient init 2024-03-29T10:38:27.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.289+0000 7fb02000e640 10 monclient: init 2024-03-29T10:38:27.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.290+0000 7fb02000e640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:27.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.290+0000 7fb02000e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:38:27.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.290+0000 7fb02000e640 10 monclient(hunting): picked mon.noname-c con 0x7fb0180b4330 addr v1:172.21.15.84:6790/0 2024-03-29T10:38:27.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.290+0000 7fb02000e640 10 monclient(hunting): picked mon.noname-b con 0x7fb0181439b0 addr v1:172.21.15.86:6789/0 2024-03-29T10:38:27.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.290+0000 7fb02000e640 10 monclient(hunting): picked mon.noname-a con 0x7fb0181470b0 addr v1:172.21.15.84:6789/0 2024-03-29T10:38:27.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.290+0000 7fb01d5a9640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fb0181439b0 0x7fb0181405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:56518/0 (socket says 172.21.15.84:56518) 2024-03-29T10:38:27.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.290+0000 7fb01d5a9640 1 -- 172.21.15.84:0/2669666711 learned_addr learned my addr 172.21.15.84:0/2669666711 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:38:27.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.290+0000 7fb02000e640 1 -- 172.21.15.84:0/2669666711 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb018121aa0 con 0x7fb0181470b0 2024-03-29T10:38:27.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.290+0000 7fb02000e640 1 -- 172.21.15.84:0/2669666711 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0181dbc10 con 0x7fb0181439b0 2024-03-29T10:38:27.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.290+0000 7fb02000e640 1 -- 172.21.15.84:0/2669666711 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0181df150 con 0x7fb0180b4330 2024-03-29T10:38:27.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.290+0000 7fb02000e640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:27.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.290+0000 7fb02000e640 10 monclient(hunting): authenticate will time out at 1009.553933s 2024-03-29T10:38:27.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.293+0000 7fb00e7fc640 10 client.?.objecter ms_handle_connect 0x7fb0181439b0 2024-03-29T10:38:27.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.293+0000 7fb00e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:27.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.293+0000 7fb00e7fc640 1 -- 172.21.15.84:0/2669666711 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1747560309 0 0) 0x7fb0181dbc10 con 0x7fb0181439b0 2024-03-29T10:38:27.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.293+0000 7fb00e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:27.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.293+0000 7fb00e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:27.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.293+0000 7fb00e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:27.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.293+0000 7fb00e7fc640 10 monclient(hunting): my global_id is 4351 2024-03-29T10:38:27.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.293+0000 7fb00e7fc640 1 -- 172.21.15.84:0/2669666711 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faffc004eb0 con 0x7fb0181439b0 2024-03-29T10:38:27.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.294+0000 7fb00e7fc640 1 -- 172.21.15.84:0/2669666711 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3358270889 0 0) 0x7faffc004eb0 con 0x7fb0181439b0 2024-03-29T10:38:27.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.294+0000 7fb00e7fc640 1 -- 172.21.15.84:0/2669666711 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0181dbc10 con 0x7fb0181439b0 2024-03-29T10:38:27.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.294+0000 7fb00e7fc640 1 -- 172.21.15.84:0/2669666711 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fb014001950 con 0x7fb0181439b0 2024-03-29T10:38:27.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.294+0000 7fb00e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:27.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.294+0000 7fb00e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:38:27.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.294+0000 7fb00e7fc640 10 monclient(hunting): dump: 2024-03-29T10:38:27.285 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:27.285 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:27.285 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:27.285 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:27.285 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:27.285 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:27.286 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:27.286 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:27.286 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:27.286 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:27.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.294+0000 7fb00e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:27.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.294+0000 7fb00e7fc640 10 client.?.objecter ms_handle_connect 0x7fb0181470b0 2024-03-29T10:38:27.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.294+0000 7fb00e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:27.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.294+0000 7fb00e7fc640 1 -- 172.21.15.84:0/2669666711 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 231437770 0 0) 0x7fb0181dbc10 con 0x7fb0181439b0 2024-03-29T10:38:27.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.294+0000 7fb00e7fc640 1 -- 172.21.15.84:0/2669666711 >> v1:172.21.15.84:6790/0 conn(0x7fb0180b4330 legacy=0x7fb0181dea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:27.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.294+0000 7fb00e7fc640 1 -- 172.21.15.84:0/2669666711 >> v1:172.21.15.84:6789/0 conn(0x7fb0181470b0 legacy=0x7fb0181e6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:27.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.294+0000 7fb00e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:38:27.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.294+0000 7fb00e7fc640 1 monclient: found mon.b 2024-03-29T10:38:27.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.294+0000 7fb00e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:27.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.294+0000 7fb00e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:38:27.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.294+0000 7fb00e7fc640 1 -- 172.21.15.84:0/2669666711 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0180b77e0 con 0x7fb0181439b0 2024-03-29T10:38:27.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.294+0000 7fb00e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:38:27.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.294+0000 7fb00e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:38:27.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.294+0000 7fb00e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:27.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.294+0000 7fb02000e640 5 monclient: authenticate success, global_id 4351 2024-03-29T10:38:27.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.294+0000 7fb02000e640 10 monclient: _renew_subs 2024-03-29T10:38:27.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.294+0000 7fb02000e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:38:27.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.294+0000 7fb02000e640 1 -- 172.21.15.84:0/2669666711 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0181dc130 con 0x7fb0181439b0 2024-03-29T10:38:27.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.295+0000 7fb02000e640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:27.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.295+0000 7fb02000e640 10 monclient: _renew_subs 2024-03-29T10:38:27.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.295+0000 7fb02000e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:38:27.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.295+0000 7fb02000e640 1 -- 172.21.15.84:0/2669666711 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb018140d00 con 0x7fb0181439b0 2024-03-29T10:38:27.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.295+0000 7fb02000e640 1 librados: init done 2024-03-29T10:38:27.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.295+0000 7fb00e7fc640 1 -- 172.21.15.84:0/2669666711 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb014001ab0 con 0x7fb0181439b0 2024-03-29T10:38:27.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.295+0000 7fb00e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:27.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.295+0000 7fb00ffff640 4 set_mon_vals no callback set 2024-03-29T10:38:27.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.296+0000 7fb00e7fc640 1 -- 172.21.15.84:0/2669666711 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fb014004c30 con 0x7fb0181439b0 2024-03-29T10:38:27.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.296+0000 7fb00e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:27.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.296+0000 7fb00e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:27.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.296+0000 7fb00e7fc640 10 monclient: dump: 2024-03-29T10:38:27.288 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:27.288 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:27.288 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:27.288 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:27.288 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:27.288 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:27.288 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:27.288 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:27.288 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:27.289 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:27.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.296+0000 7faffbfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:27.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.297+0000 7faffbfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:27.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.297+0000 7faffbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:38:27.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.297+0000 7faffbfff640 1 -- 172.21.15.84:0/2669666711 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0181dbda0 con 0x7fb0181439b0 2024-03-29T10:38:27.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.297+0000 7fb00e7fc640 1 -- 172.21.15.84:0/2669666711 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7fb014004d90 con 0x7fb0181439b0 2024-03-29T10:38:27.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.298+0000 7fb00e7fc640 1 -- 172.21.15.84:0/2669666711 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7fb014084b30 con 0x7fb0181439b0 2024-03-29T10:38:27.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.298+0000 7fb00e7fc640 10 client.4351.objecter ms_dispatch 0x7fb018000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:38:27.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.298+0000 7fb00e7fc640 3 client.4351.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:38:27.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.298+0000 7fb00e7fc640 3 client.4351.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:38:27.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.298+0000 7fb00e7fc640 20 client.4351.objecter dump_active .. 0 homeless 2024-03-29T10:38:27.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.298+0000 7fb01ddaa640 10 monclient: get_auth_request con 0x7faffc06dc10 auth_method 0 2024-03-29T10:38:27.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.299+0000 7fb00e7fc640 10 client.4351.objecter ms_handle_connect 0x7faffc06dc10 2024-03-29T10:38:27.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.306+0000 7fb00e7fc640 1 -- 172.21.15.84:0/2669666711 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb014050410 con 0x7fb0181439b0 2024-03-29T10:38:27.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.306+0000 7fb00e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:27.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.306+0000 7fb00e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:27.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.624+0000 7faffbfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:27.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.624+0000 7faffbfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:27.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.624+0000 7faffbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:38:27.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.624+0000 7faffbfff640 1 -- 172.21.15.84:0/2669666711 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb018141050 con 0x7fb0181439b0 2024-03-29T10:38:27.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.625+0000 7fb00e7fc640 1 -- 172.21.15.84:0/2669666711 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7fb014052340 con 0x7fb0181439b0 2024-03-29T10:38:27.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.625+0000 7fb00e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:27.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.625+0000 7fb00e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:27.610 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:38:27.610 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:38:27.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.629+0000 7faffbfff640 10 librados: watch_flush enter 2024-03-29T10:38:27.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.629+0000 7faffbfff640 10 librados: watch_flush exit 2024-03-29T10:38:27.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.629+0000 7faffbfff640 20 client.4351.objecter shutdown clearing up homeless session... 2024-03-29T10:38:27.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.629+0000 7faffbfff640 10 client.4351.objecter successfully canceled tick 2024-03-29T10:38:27.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.629+0000 7faffbfff640 1 -- 172.21.15.84:0/2669666711 >> v1:172.21.15.86:6816/3337225337 conn(0x7faffc06dc10 legacy=0x7faffc0700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:27.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.629+0000 7faffbfff640 10 monclient: shutdown 2024-03-29T10:38:27.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.629+0000 7faffbfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:27.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.629+0000 7faffbfff640 1 -- 172.21.15.84:0/2669666711 >> v1:172.21.15.86:6789/0 conn(0x7fb0181439b0 legacy=0x7fb0181405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:27.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.629+0000 7faffbfff640 1 -- 172.21.15.84:0/2669666711 shutdown_connections 2024-03-29T10:38:27.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.629+0000 7faffbfff640 1 -- 172.21.15.84:0/2669666711 >> 172.21.15.84:0/2669666711 conn(0x7fb018066a00 msgr2=0x7fb0181d1390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:27.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.629+0000 7faffbfff640 1 -- 172.21.15.84:0/2669666711 shutdown_connections 2024-03-29T10:38:27.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.629+0000 7faffbfff640 1 -- 172.21.15.84:0/2669666711 wait complete. 2024-03-29T10:38:27.615 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:27.629+0000 7faffbfff640 1 librados: shutdown 2024-03-29T10:38:27.626 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:38:34.433 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:38:34.582 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.598+0000 7f4a30b3b640 10 monclient: get_monmap_and_config 2024-03-29T10:38:34.583 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.598+0000 7f4a30b3b640 10 monclient: build_initial_monmap 2024-03-29T10:38:34.583 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.598+0000 7f4a30b3b640 10 monclient: monmap: 2024-03-29T10:38:34.583 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:38:34.583 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:34.583 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:38:34.599477+0000 2024-03-29T10:38:34.583 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:38:34.599477+0000 2024-03-29T10:38:34.583 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:34.583 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:34.583 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:38:34.583 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:38:34.583 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:38:34.583 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:34.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.599+0000 7f4a30b3b640 1 Processor -- start 2024-03-29T10:38:34.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.599+0000 7f4a30b3b640 1 -- start start 2024-03-29T10:38:34.584 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.599+0000 7f4a30b3b640 10 monclient: init 2024-03-29T10:38:34.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.600+0000 7f4a30b3b640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:34.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.600+0000 7f4a30b3b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:38:34.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.600+0000 7f4a30b3b640 10 monclient(hunting): picked mon.noname-b con 0x7f4a2c149ed0 addr v1:172.21.15.84:6790/0 2024-03-29T10:38:34.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.600+0000 7f4a30b3b640 10 monclient(hunting): picked mon.noname-a con 0x7f4a2c072370 addr v1:172.21.15.84:6789/0 2024-03-29T10:38:34.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.600+0000 7f4a30b3b640 10 monclient(hunting): picked mon.noname-c con 0x7f4a2c14a2b0 addr v1:172.21.15.86:6789/0 2024-03-29T10:38:34.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.600+0000 7f4a30b3b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a2c14aa50 con 0x7f4a2c072370 2024-03-29T10:38:34.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.600+0000 7f4a30b3b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a2c14a760 con 0x7f4a2c14a2b0 2024-03-29T10:38:34.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.600+0000 7f4a30b3b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a2c070190 con 0x7f4a2c149ed0 2024-03-29T10:38:34.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.600+0000 7f4a30b3b640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:34.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.600+0000 7f4a30b3b640 10 monclient(hunting): authenticate will time out at 1016.863884s 2024-03-29T10:38:34.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.602+0000 7f4a2a59c640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f4a2c149ed0 0x7f4a2c071c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:52094/0 (socket says 172.21.15.84:52094) 2024-03-29T10:38:34.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.602+0000 7f4a2a59c640 1 -- 172.21.15.84:0/4200435102 learned_addr learned my addr 172.21.15.84:0/4200435102 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:38:34.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.603+0000 7f4a2959a640 1 -- 172.21.15.84:0/4200435102 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1441962501 0 0) 0x7f4a2c070190 con 0x7f4a2c149ed0 2024-03-29T10:38:34.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.603+0000 7f4a2959a640 10 monclient(hunting): _negotiate 2024-03-29T10:38:34.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.603+0000 7f4a2959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:34.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.603+0000 7f4a2959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:34.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.603+0000 7f4a2959a640 10 monclient(hunting): my global_id is 4388 2024-03-29T10:38:34.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.603+0000 7f4a2959a640 1 -- 172.21.15.84:0/4200435102 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a14003520 con 0x7f4a2c149ed0 2024-03-29T10:38:34.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.604+0000 7f4a2959a640 1 -- 172.21.15.84:0/4200435102 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1736504987 0 0) 0x7f4a14003520 con 0x7f4a2c149ed0 2024-03-29T10:38:34.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.604+0000 7f4a2959a640 1 -- 172.21.15.84:0/4200435102 >> v1:172.21.15.86:6789/0 conn(0x7f4a2c14a2b0 legacy=0x7f4a2c06c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:38:34.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.604+0000 7f4a2959a640 1 -- 172.21.15.84:0/4200435102 >> v1:172.21.15.84:6789/0 conn(0x7f4a2c072370 legacy=0x7f4a2c06c150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:38:34.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.604+0000 7f4a2959a640 10 monclient: _finish_hunting 0 2024-03-29T10:38:34.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.604+0000 7f4a2959a640 1 monclient: found mon.noname-b 2024-03-29T10:38:34.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.604+0000 7f4a2959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:34.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.604+0000 7f4a2959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:38:34.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.604+0000 7f4a2959a640 1 -- 172.21.15.84:0/4200435102 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2c070470 con 0x7f4a2c149ed0 2024-03-29T10:38:34.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.604+0000 7f4a2959a640 10 monclient: _finish_auth 0 2024-03-29T10:38:34.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.604+0000 7f4a2959a640 10 monclient: _check_auth_tickets 2024-03-29T10:38:34.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.604+0000 7f4a2959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:34.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.604+0000 7f4a2959a640 1 -- 172.21.15.84:0/4200435102 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4a24001a10 con 0x7f4a2c149ed0 2024-03-29T10:38:34.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.604+0000 7f4a2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:34.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.604+0000 7f4a2959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:38:34.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.604+0000 7f4a2959a640 10 monclient: dump: 2024-03-29T10:38:34.589 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:34.589 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:34.590 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:34.590 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:34.590 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:34.590 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:34.590 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:34.590 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:34.590 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:34.590 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:34.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.604+0000 7f4a30b3b640 5 monclient: authenticate success, global_id 4388 2024-03-29T10:38:34.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.604+0000 7f4a30b3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:34.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.604+0000 7f4a2959a640 1 -- 172.21.15.84:0/4200435102 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a24001b70 con 0x7f4a2c149ed0 2024-03-29T10:38:34.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.604+0000 7f4a2959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:34.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.604+0000 7f4a30b3b640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:34.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.604+0000 7f4a30b3b640 4 set_mon_vals no callback set 2024-03-29T10:38:34.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.605+0000 7f4a30b3b640 10 monclient: shutdown 2024-03-29T10:38:34.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.605+0000 7f4a30b3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:34.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.605+0000 7f4a30b3b640 1 -- 172.21.15.84:0/4200435102 >> v1:172.21.15.84:6790/0 conn(0x7f4a2c149ed0 legacy=0x7f4a2c071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:34.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.607+0000 7f4a30b3b640 1 -- 172.21.15.84:0/4200435102 shutdown_connections 2024-03-29T10:38:34.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.607+0000 7f4a30b3b640 1 -- 172.21.15.84:0/4200435102 >> 172.21.15.84:0/4200435102 conn(0x7f4a2c0675b0 msgr2=0x7f4a2c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:34.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.609+0000 7f4a30b3b640 1 -- 172.21.15.84:0/4200435102 shutdown_connections 2024-03-29T10:38:34.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.610+0000 7f4a30b3b640 1 -- 172.21.15.84:0/4200435102 wait complete. 2024-03-29T10:38:34.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.612+0000 7f4a30b3b640 10 monclient: build_initial_monmap 2024-03-29T10:38:34.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.612+0000 7f4a30b3b640 10 monclient: monmap: 2024-03-29T10:38:34.597 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:38:34.597 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:34.597 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:38:34.597 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:38:34.597 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:34.597 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:34.597 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:38:34.597 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:38:34.598 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:38:34.598 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:34.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.612+0000 7f4a30b3b640 1 librados: starting msgr at 2024-03-29T10:38:34.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.612+0000 7f4a30b3b640 1 librados: starting objecter 2024-03-29T10:38:34.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.612+0000 7f4a30b3b640 1 Processor -- start 2024-03-29T10:38:34.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.612+0000 7f4a30b3b640 1 -- start start 2024-03-29T10:38:34.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.612+0000 7f4a30b3b640 1 librados: setting wanted keys 2024-03-29T10:38:34.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.612+0000 7f4a30b3b640 1 librados: calling monclient init 2024-03-29T10:38:34.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.612+0000 7f4a30b3b640 10 monclient: init 2024-03-29T10:38:34.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.613+0000 7f4a30b3b640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:34.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.613+0000 7f4a30b3b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:38:34.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.613+0000 7f4a30b3b640 10 monclient(hunting): picked mon.noname-a con 0x7f4a2c072370 addr v1:172.21.15.84:6789/0 2024-03-29T10:38:34.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.613+0000 7f4a30b3b640 10 monclient(hunting): picked mon.noname-c con 0x7f4a2c149ed0 addr v1:172.21.15.84:6790/0 2024-03-29T10:38:34.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.613+0000 7f4a30b3b640 10 monclient(hunting): picked mon.noname-b con 0x7f4a2c14a2b0 addr v1:172.21.15.86:6789/0 2024-03-29T10:38:34.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.613+0000 7f4a30b3b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a2c0af5c0 con 0x7f4a2c072370 2024-03-29T10:38:34.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.613+0000 7f4a30b3b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a2c11dd60 con 0x7f4a2c14a2b0 2024-03-29T10:38:34.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.613+0000 7f4a30b3b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a2c0af1e0 con 0x7f4a2c149ed0 2024-03-29T10:38:34.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.613+0000 7f4a30b3b640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:34.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.613+0000 7f4a30b3b640 10 monclient(hunting): authenticate will time out at 1016.876658s 2024-03-29T10:38:34.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.613+0000 7f4a29d9b640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f4a2c149ed0 0x7f4a2c143870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:54508/0 (socket says 172.21.15.84:54508) 2024-03-29T10:38:34.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.613+0000 7f4a29d9b640 1 -- 172.21.15.84:0/2779657135 learned_addr learned my addr 172.21.15.84:0/2779657135 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:38:34.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.613+0000 7f4a1affd640 10 client.?.objecter ms_handle_connect 0x7f4a2c149ed0 2024-03-29T10:38:34.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.613+0000 7f4a1affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:34.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.614+0000 7f4a1affd640 1 -- 172.21.15.84:0/2779657135 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159135719 0 0) 0x7f4a2c0af1e0 con 0x7f4a2c149ed0 2024-03-29T10:38:34.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.614+0000 7f4a1affd640 10 monclient(hunting): _negotiate 2024-03-29T10:38:34.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.614+0000 7f4a1affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:34.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.614+0000 7f4a1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:34.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.614+0000 7f4a1affd640 10 monclient(hunting): my global_id is 4391 2024-03-29T10:38:34.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.614+0000 7f4a1affd640 1 -- 172.21.15.84:0/2779657135 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a0c004eb0 con 0x7f4a2c149ed0 2024-03-29T10:38:34.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.614+0000 7f4a1affd640 1 -- 172.21.15.84:0/2779657135 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 294244642 0 0) 0x7f4a0c004eb0 con 0x7f4a2c149ed0 2024-03-29T10:38:34.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.614+0000 7f4a1affd640 1 -- 172.21.15.84:0/2779657135 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a2c0af1e0 con 0x7f4a2c149ed0 2024-03-29T10:38:34.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.614+0000 7f4a1affd640 1 -- 172.21.15.84:0/2779657135 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4a1c001930 con 0x7f4a2c149ed0 2024-03-29T10:38:34.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.614+0000 7f4a1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:34.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.614+0000 7f4a1affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:38:34.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.614+0000 7f4a1affd640 10 monclient(hunting): dump: 2024-03-29T10:38:34.606 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:34.606 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:34.606 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:34.606 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:34.606 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:34.606 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:34.606 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:34.606 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:34.606 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:34.606 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:34.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.614+0000 7f4a1affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:34.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.614+0000 7f4a1affd640 1 -- 172.21.15.84:0/2779657135 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3683451291 0 0) 0x7f4a2c0af1e0 con 0x7f4a2c149ed0 2024-03-29T10:38:34.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.614+0000 7f4a1affd640 1 -- 172.21.15.84:0/2779657135 >> v1:172.21.15.86:6789/0 conn(0x7f4a2c14a2b0 legacy=0x7f4a2c0aceb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:38:34.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.617+0000 7f4a1affd640 1 -- 172.21.15.84:0/2779657135 >> v1:172.21.15.84:6789/0 conn(0x7f4a2c072370 legacy=0x7f4a2c0aead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:34.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.617+0000 7f4a1affd640 10 monclient: _finish_hunting 0 2024-03-29T10:38:34.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.617+0000 7f4a1affd640 1 monclient: found mon.c 2024-03-29T10:38:34.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.617+0000 7f4a1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:34.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.617+0000 7f4a1affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:38:34.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.617+0000 7f4a1affd640 1 -- 172.21.15.84:0/2779657135 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a2c143f80 con 0x7f4a2c149ed0 2024-03-29T10:38:34.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.617+0000 7f4a1affd640 10 monclient: _finish_auth 0 2024-03-29T10:38:34.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.617+0000 7f4a1affd640 10 monclient: _check_auth_tickets 2024-03-29T10:38:34.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.617+0000 7f4a1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:34.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.617+0000 7f4a1affd640 10 client.?.objecter ms_handle_connect 0x7f4a2c072370 2024-03-29T10:38:34.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.617+0000 7f4a1affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:34.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.617+0000 7f4a1affd640 1 -- 172.21.15.84:0/2779657135 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a1c001c50 con 0x7f4a2c149ed0 2024-03-29T10:38:34.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.617+0000 7f4a1affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:34.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.617+0000 7f4a1affd640 1 -- 172.21.15.84:0/2779657135 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4a1c003a40 con 0x7f4a2c149ed0 2024-03-29T10:38:34.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.617+0000 7f4a1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:34.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.617+0000 7f4a1affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:34.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.617+0000 7f4a1affd640 10 monclient: dump: 2024-03-29T10:38:34.608 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:34.608 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:34.608 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:34.608 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:34.608 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:34.608 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:34.608 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:34.608 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:34.608 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:34.608 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:34.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.618+0000 7f4a28d99640 4 set_mon_vals no callback set 2024-03-29T10:38:34.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.619+0000 7f4a30b3b640 5 monclient: authenticate success, global_id 4391 2024-03-29T10:38:34.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.619+0000 7f4a30b3b640 10 monclient: _renew_subs 2024-03-29T10:38:34.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.619+0000 7f4a30b3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:38:34.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.619+0000 7f4a30b3b640 1 -- 172.21.15.84:0/2779657135 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a2c1441f0 con 0x7f4a2c149ed0 2024-03-29T10:38:34.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.619+0000 7f4a30b3b640 10 client.4391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:34.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.619+0000 7f4a30b3b640 10 monclient: _renew_subs 2024-03-29T10:38:34.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.619+0000 7f4a30b3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:38:34.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.619+0000 7f4a30b3b640 1 -- 172.21.15.84:0/2779657135 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a2c0ab710 con 0x7f4a2c149ed0 2024-03-29T10:38:34.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.619+0000 7f4a30b3b640 1 librados: init done 2024-03-29T10:38:34.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.621+0000 7f4a30b3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:34.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.621+0000 7f4a30b3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:34.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.621+0000 7f4a1affd640 1 -- 172.21.15.84:0/2779657135 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f4a1c003ba0 con 0x7f4a2c149ed0 2024-03-29T10:38:34.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.621+0000 7f4a30b3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:38:34.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.621+0000 7f4a30b3b640 1 -- 172.21.15.84:0/2779657135 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a2c075f10 con 0x7f4a2c149ed0 2024-03-29T10:38:34.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.621+0000 7f4a1affd640 1 -- 172.21.15.84:0/2779657135 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f4a1c084af0 con 0x7f4a2c149ed0 2024-03-29T10:38:34.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.621+0000 7f4a1affd640 10 client.4391.objecter ms_dispatch 0x7f4a2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:38:34.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.621+0000 7f4a1affd640 3 client.4391.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:38:34.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.621+0000 7f4a1affd640 3 client.4391.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:38:34.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.622+0000 7f4a1affd640 20 client.4391.objecter dump_active .. 0 homeless 2024-03-29T10:38:34.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.622+0000 7f4a2a59c640 10 monclient: get_auth_request con 0x7f4a0c06dc10 auth_method 0 2024-03-29T10:38:34.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.622+0000 7f4a1affd640 10 client.4391.objecter ms_handle_connect 0x7f4a0c06dc10 2024-03-29T10:38:34.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.628+0000 7f4a1affd640 1 -- 172.21.15.84:0/2779657135 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4a1c0503d0 con 0x7f4a2c149ed0 2024-03-29T10:38:34.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.628+0000 7f4a1affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:34.614 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.628+0000 7f4a1affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:34.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.942+0000 7f4a30b3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:34.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.942+0000 7f4a30b3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:34.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.942+0000 7f4a30b3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:38:34.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.942+0000 7f4a30b3b640 1 -- 172.21.15.84:0/2779657135 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a2c147c80 con 0x7f4a2c149ed0 2024-03-29T10:38:34.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.943+0000 7f4a1affd640 1 -- 172.21.15.84:0/2779657135 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f4a1c052300 con 0x7f4a2c149ed0 2024-03-29T10:38:34.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.943+0000 7f4a1affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:34.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.943+0000 7f4a1affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:34.928 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:38:34.928 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:38:34.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.946+0000 7f4a30b3b640 10 librados: watch_flush enter 2024-03-29T10:38:34.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.947+0000 7f4a30b3b640 10 librados: watch_flush exit 2024-03-29T10:38:34.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.947+0000 7f4a30b3b640 20 client.4391.objecter shutdown clearing up homeless session... 2024-03-29T10:38:34.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.947+0000 7f4a30b3b640 10 client.4391.objecter successfully canceled tick 2024-03-29T10:38:34.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.947+0000 7f4a30b3b640 1 -- 172.21.15.84:0/2779657135 >> v1:172.21.15.86:6816/3337225337 conn(0x7f4a0c06dc10 legacy=0x7f4a0c0700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:34.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.947+0000 7f4a30b3b640 10 monclient: shutdown 2024-03-29T10:38:34.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.947+0000 7f4a30b3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:34.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.947+0000 7f4a30b3b640 1 -- 172.21.15.84:0/2779657135 >> v1:172.21.15.84:6790/0 conn(0x7f4a2c149ed0 legacy=0x7f4a2c143870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:34.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.947+0000 7f4a30b3b640 1 -- 172.21.15.84:0/2779657135 shutdown_connections 2024-03-29T10:38:34.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.947+0000 7f4a30b3b640 1 -- 172.21.15.84:0/2779657135 >> 172.21.15.84:0/2779657135 conn(0x7f4a2c0675b0 msgr2=0x7f4a2c0b2c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:34.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.948+0000 7f4a30b3b640 1 -- 172.21.15.84:0/2779657135 shutdown_connections 2024-03-29T10:38:34.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.948+0000 7f4a30b3b640 1 -- 172.21.15.84:0/2779657135 wait complete. 2024-03-29T10:38:34.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:34.948+0000 7f4a30b3b640 1 librados: shutdown 2024-03-29T10:38:34.949 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:38:41.755 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:38:41.902 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.917+0000 7f37b5a4a640 10 monclient: get_monmap_and_config 2024-03-29T10:38:41.902 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.917+0000 7f37b5a4a640 10 monclient: build_initial_monmap 2024-03-29T10:38:41.902 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.917+0000 7f37b5a4a640 10 monclient: monmap: 2024-03-29T10:38:41.902 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:38:41.902 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:41.902 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:38:41.919053+0000 2024-03-29T10:38:41.902 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:38:41.919053+0000 2024-03-29T10:38:41.903 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:41.903 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:41.903 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:38:41.903 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:38:41.903 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:38:41.903 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:41.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.918+0000 7f37b5a4a640 1 Processor -- start 2024-03-29T10:38:41.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.918+0000 7f37b5a4a640 1 -- start start 2024-03-29T10:38:41.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.918+0000 7f37b5a4a640 10 monclient: init 2024-03-29T10:38:41.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.918+0000 7f37b5a4a640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:41.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.918+0000 7f37b5a4a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:38:41.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.918+0000 7f37b5a4a640 10 monclient(hunting): picked mon.noname-a con 0x7f37b01422b0 addr v1:172.21.15.84:6789/0 2024-03-29T10:38:41.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.918+0000 7f37b5a4a640 10 monclient(hunting): picked mon.noname-c con 0x7f37b0142690 addr v1:172.21.15.86:6789/0 2024-03-29T10:38:41.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.918+0000 7f37b5a4a640 10 monclient(hunting): picked mon.noname-b con 0x7f37b00b17d0 addr v1:172.21.15.84:6790/0 2024-03-29T10:38:41.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.918+0000 7f37b5a4a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b0142c80 con 0x7f37b01422b0 2024-03-29T10:38:41.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.918+0000 7f37b5a4a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b00aae40 con 0x7f37b0142690 2024-03-29T10:38:41.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.918+0000 7f37b5a4a640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b013fbd0 con 0x7f37b00b17d0 2024-03-29T10:38:41.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.918+0000 7f37b5a4a640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:41.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.918+0000 7f37b5a4a640 10 monclient(hunting): authenticate will time out at 1024.183073s 2024-03-29T10:38:41.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.919+0000 7f37af7fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f37b00b17d0 0x7f37b00b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:44822/0 (socket says 172.21.15.84:44822) 2024-03-29T10:38:41.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.919+0000 7f37af7fe640 1 -- 172.21.15.84:0/533329424 learned_addr learned my addr 172.21.15.84:0/533329424 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:38:41.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.920+0000 7f37adffb640 1 -- 172.21.15.84:0/533329424 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2951381885 0 0) 0x7f37b013fbd0 con 0x7f37b00b17d0 2024-03-29T10:38:41.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.920+0000 7f37adffb640 10 monclient(hunting): _negotiate 2024-03-29T10:38:41.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.920+0000 7f37adffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:41.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.920+0000 7f37adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:41.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.920+0000 7f37adffb640 10 monclient(hunting): my global_id is 4394 2024-03-29T10:38:41.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.920+0000 7f37adffb640 1 -- 172.21.15.84:0/533329424 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3798003520 con 0x7f37b00b17d0 2024-03-29T10:38:41.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.921+0000 7f37adffb640 1 -- 172.21.15.84:0/533329424 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3040125229 0 0) 0x7f3798003520 con 0x7f37b00b17d0 2024-03-29T10:38:41.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.921+0000 7f37adffb640 1 -- 172.21.15.84:0/533329424 >> v1:172.21.15.86:6789/0 conn(0x7f37b0142690 legacy=0x7f37b00b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:38:41.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.923+0000 7f37adffb640 1 -- 172.21.15.84:0/533329424 >> v1:172.21.15.84:6789/0 conn(0x7f37b01422b0 legacy=0x7f37b013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:38:41.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.923+0000 7f37adffb640 10 monclient: _finish_hunting 0 2024-03-29T10:38:41.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.923+0000 7f37adffb640 1 monclient: found mon.noname-b 2024-03-29T10:38:41.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.923+0000 7f37adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:41.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.923+0000 7f37adffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:38:41.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.923+0000 7f37adffb640 1 -- 172.21.15.84:0/533329424 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b0151940 con 0x7f37b00b17d0 2024-03-29T10:38:41.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.923+0000 7f37adffb640 10 monclient: _finish_auth 0 2024-03-29T10:38:41.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.923+0000 7f37adffb640 10 monclient: _check_auth_tickets 2024-03-29T10:38:41.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.923+0000 7f37adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:41.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.923+0000 7f37b5a4a640 5 monclient: authenticate success, global_id 4394 2024-03-29T10:38:41.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.923+0000 7f37b5a4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:41.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.923+0000 7f37adffb640 1 -- 172.21.15.84:0/533329424 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f37a0001690 con 0x7f37b00b17d0 2024-03-29T10:38:41.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.923+0000 7f37adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:41.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.923+0000 7f37adffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:38:41.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.923+0000 7f37adffb640 10 monclient: dump: 2024-03-29T10:38:41.912 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:41.912 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:41.912 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:41.912 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:41.912 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:41.912 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:41.912 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:41.912 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:41.912 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:41.912 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:41.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.923+0000 7f37b5a4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:41.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.923+0000 7f37adffb640 1 -- 172.21.15.84:0/533329424 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37a00017f0 con 0x7f37b00b17d0 2024-03-29T10:38:41.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.923+0000 7f37adffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:41.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.923+0000 7f37b5a4a640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:41.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.923+0000 7f37b5a4a640 4 set_mon_vals no callback set 2024-03-29T10:38:41.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.924+0000 7f37adffb640 1 -- 172.21.15.84:0/533329424 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f37a0001c00 con 0x7f37b00b17d0 2024-03-29T10:38:41.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.924+0000 7f37adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:41.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.924+0000 7f37adffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:41.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.924+0000 7f37adffb640 10 monclient: dump: 2024-03-29T10:38:41.913 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:41.913 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:41.913 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:41.913 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:41.913 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:41.913 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:41.913 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:41.913 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:41.913 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:41.913 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:41.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.924+0000 7f37b5a4a640 10 monclient: shutdown 2024-03-29T10:38:41.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.924+0000 7f37b5a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:41.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.924+0000 7f37b5a4a640 1 -- 172.21.15.84:0/533329424 >> v1:172.21.15.84:6790/0 conn(0x7f37b00b17d0 legacy=0x7f37b00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:41.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.924+0000 7f37b5a4a640 1 -- 172.21.15.84:0/533329424 shutdown_connections 2024-03-29T10:38:41.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.924+0000 7f37b5a4a640 1 -- 172.21.15.84:0/533329424 >> 172.21.15.84:0/533329424 conn(0x7f37b0066a00 msgr2=0x7f37b0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:41.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.924+0000 7f37b5a4a640 1 -- 172.21.15.84:0/533329424 shutdown_connections 2024-03-29T10:38:41.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.924+0000 7f37b5a4a640 1 -- 172.21.15.84:0/533329424 wait complete. 2024-03-29T10:38:41.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.925+0000 7f37b5a4a640 10 monclient: build_initial_monmap 2024-03-29T10:38:41.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.925+0000 7f37b5a4a640 10 monclient: monmap: 2024-03-29T10:38:41.916 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:38:41.916 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:41.916 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:38:41.916 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:38:41.916 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:41.916 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:41.916 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:38:41.916 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:38:41.916 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:38:41.916 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:41.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.926+0000 7f37b5a4a640 1 librados: starting msgr at 2024-03-29T10:38:41.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.926+0000 7f37b5a4a640 1 librados: starting objecter 2024-03-29T10:38:41.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.926+0000 7f37b5a4a640 1 Processor -- start 2024-03-29T10:38:41.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.927+0000 7f37b5a4a640 1 -- start start 2024-03-29T10:38:41.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.927+0000 7f37b5a4a640 1 librados: setting wanted keys 2024-03-29T10:38:41.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.927+0000 7f37b5a4a640 1 librados: calling monclient init 2024-03-29T10:38:41.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.927+0000 7f37b5a4a640 10 monclient: init 2024-03-29T10:38:41.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.928+0000 7f37b5a4a640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:41.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.928+0000 7f37b5a4a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:38:41.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.928+0000 7f37b5a4a640 10 monclient(hunting): picked mon.noname-b con 0x7f37b00b17d0 addr v1:172.21.15.86:6789/0 2024-03-29T10:38:41.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.928+0000 7f37b5a4a640 10 monclient(hunting): picked mon.noname-a con 0x7f37b01422b0 addr v1:172.21.15.84:6789/0 2024-03-29T10:38:41.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.928+0000 7f37b5a4a640 10 monclient(hunting): picked mon.noname-c con 0x7f37b0142690 addr v1:172.21.15.84:6790/0 2024-03-29T10:38:41.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.928+0000 7f37b5a4a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b01daba0 con 0x7f37b01422b0 2024-03-29T10:38:41.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.928+0000 7f37b5a4a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b00ad240 con 0x7f37b00b17d0 2024-03-29T10:38:41.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.928+0000 7f37b5a4a640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37b00ac0e0 con 0x7f37b0142690 2024-03-29T10:38:41.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.928+0000 7f37b5a4a640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:41.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.928+0000 7f37b5a4a640 10 monclient(hunting): authenticate will time out at 1024.192722s 2024-03-29T10:38:41.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.932+0000 7f37af7fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f37b0142690 0x7f37b00b0f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:47122/0 (socket says 172.21.15.84:47122) 2024-03-29T10:38:41.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.932+0000 7f37af7fe640 1 -- 172.21.15.84:0/1983269130 learned_addr learned my addr 172.21.15.84:0/1983269130 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:38:41.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.932+0000 7f3797fff640 10 client.?.objecter ms_handle_connect 0x7f37b0142690 2024-03-29T10:38:41.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.932+0000 7f3797fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:41.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.932+0000 7f3797fff640 10 client.?.objecter ms_handle_connect 0x7f37b00b17d0 2024-03-29T10:38:41.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.932+0000 7f3797fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:41.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.932+0000 7f3797fff640 1 -- 172.21.15.84:0/1983269130 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 225362048 0 0) 0x7f37b00ac0e0 con 0x7f37b0142690 2024-03-29T10:38:41.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.932+0000 7f3797fff640 10 monclient(hunting): _negotiate 2024-03-29T10:38:41.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.932+0000 7f3797fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:41.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.932+0000 7f3797fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:41.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.932+0000 7f3797fff640 10 monclient(hunting): my global_id is 4397 2024-03-29T10:38:41.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.932+0000 7f3797fff640 1 -- 172.21.15.84:0/1983269130 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f379c004eb0 con 0x7f37b0142690 2024-03-29T10:38:41.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.933+0000 7f3797fff640 1 -- 172.21.15.84:0/1983269130 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2129305717 0 0) 0x7f37b00ad240 con 0x7f37b00b17d0 2024-03-29T10:38:41.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.933+0000 7f3797fff640 10 monclient(hunting): _negotiate 2024-03-29T10:38:41.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.933+0000 7f3797fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:41.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.933+0000 7f3797fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:41.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.933+0000 7f3797fff640 10 monclient(hunting): my global_id is 4354 2024-03-29T10:38:41.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.933+0000 7f3797fff640 1 -- 172.21.15.84:0/1983269130 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37b00ac0e0 con 0x7f37b00b17d0 2024-03-29T10:38:41.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.933+0000 7f3797fff640 1 -- 172.21.15.84:0/1983269130 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2529150352 0 0) 0x7f379c004eb0 con 0x7f37b0142690 2024-03-29T10:38:41.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.933+0000 7f3797fff640 1 -- 172.21.15.84:0/1983269130 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37b00ad240 con 0x7f37b0142690 2024-03-29T10:38:41.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.933+0000 7f3797fff640 1 -- 172.21.15.84:0/1983269130 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f37a0001b80 con 0x7f37b0142690 2024-03-29T10:38:41.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.933+0000 7f3797fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:41.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.933+0000 7f3797fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:38:41.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.933+0000 7f3797fff640 10 monclient(hunting): dump: 2024-03-29T10:38:41.922 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:41.922 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:41.923 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:41.923 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:41.923 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:41.923 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:41.923 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:41.923 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:41.923 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:41.923 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:41.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.933+0000 7f3797fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:41.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.933+0000 7f3797fff640 10 client.?.objecter ms_handle_connect 0x7f37b01422b0 2024-03-29T10:38:41.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.933+0000 7f3797fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:41.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.933+0000 7f3797fff640 1 -- 172.21.15.84:0/1983269130 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2646466734 0 0) 0x7f37b00ad240 con 0x7f37b0142690 2024-03-29T10:38:41.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.933+0000 7f3797fff640 1 -- 172.21.15.84:0/1983269130 >> v1:172.21.15.86:6789/0 conn(0x7f37b00b17d0 legacy=0x7f37b01db060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:41.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.933+0000 7f3797fff640 1 -- 172.21.15.84:0/1983269130 >> v1:172.21.15.84:6789/0 conn(0x7f37b01422b0 legacy=0x7f37b00b0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:41.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.933+0000 7f3797fff640 10 monclient: _finish_hunting 0 2024-03-29T10:38:41.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.933+0000 7f3797fff640 1 monclient: found mon.c 2024-03-29T10:38:41.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.933+0000 7f3797fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:41.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.933+0000 7f3797fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:38:41.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.933+0000 7f3797fff640 1 -- 172.21.15.84:0/1983269130 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b01db770 con 0x7f37b0142690 2024-03-29T10:38:41.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.933+0000 7f3797fff640 10 monclient: _finish_auth 0 2024-03-29T10:38:41.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.933+0000 7f3797fff640 10 monclient: _check_auth_tickets 2024-03-29T10:38:41.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.933+0000 7f3797fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:41.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.934+0000 7f37b5a4a640 5 monclient: authenticate success, global_id 4397 2024-03-29T10:38:41.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.934+0000 7f37b5a4a640 10 monclient: _renew_subs 2024-03-29T10:38:41.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.934+0000 7f37b5a4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:38:41.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.934+0000 7f37b5a4a640 1 -- 172.21.15.84:0/1983269130 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37b012fd30 con 0x7f37b0142690 2024-03-29T10:38:41.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.934+0000 7f37b5a4a640 10 client.4397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:41.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.934+0000 7f37b5a4a640 10 monclient: _renew_subs 2024-03-29T10:38:41.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.934+0000 7f37b5a4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:38:41.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.934+0000 7f37b5a4a640 1 -- 172.21.15.84:0/1983269130 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f37b00abd50 con 0x7f37b0142690 2024-03-29T10:38:41.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.934+0000 7f37b5a4a640 1 librados: init done 2024-03-29T10:38:41.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.935+0000 7f3797fff640 1 -- 172.21.15.84:0/1983269130 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37a0004a90 con 0x7f37b0142690 2024-03-29T10:38:41.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.936+0000 7f3797fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:41.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.936+0000 7f3797fff640 1 -- 172.21.15.84:0/1983269130 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f37a0001270 con 0x7f37b0142690 2024-03-29T10:38:41.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.936+0000 7f3797fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:41.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.936+0000 7f3797fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:41.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.936+0000 7f3797fff640 10 monclient: dump: 2024-03-29T10:38:41.928 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:41.928 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:41.928 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:41.928 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:41.928 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:41.928 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:41.928 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:41.928 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:41.928 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:41.928 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:41.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.937+0000 7f3797fff640 1 -- 172.21.15.84:0/1983269130 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f37a001ac50 con 0x7f37b0142690 2024-03-29T10:38:41.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.937+0000 7f37b5a4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:41.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.937+0000 7f37b5a4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:41.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.937+0000 7f37b5a4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:38:41.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.937+0000 7f37acff9640 4 set_mon_vals no callback set 2024-03-29T10:38:41.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.937+0000 7f37b5a4a640 1 -- 172.21.15.84:0/1983269130 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37b0075f10 con 0x7f37b0142690 2024-03-29T10:38:41.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.938+0000 7f3797fff640 1 -- 172.21.15.84:0/1983269130 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f37a0084ba0 con 0x7f37b0142690 2024-03-29T10:38:41.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.938+0000 7f3797fff640 10 client.4397.objecter ms_dispatch 0x7f37b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:38:41.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.938+0000 7f3797fff640 3 client.4397.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:38:41.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.938+0000 7f3797fff640 3 client.4397.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:38:41.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.938+0000 7f3797fff640 20 client.4397.objecter dump_active .. 0 homeless 2024-03-29T10:38:41.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.939+0000 7f37aeffd640 10 monclient: get_auth_request con 0x7f379c06ddb0 auth_method 0 2024-03-29T10:38:41.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.942+0000 7f3797fff640 10 client.4397.objecter ms_handle_connect 0x7f379c06ddb0 2024-03-29T10:38:41.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.946+0000 7f3797fff640 1 -- 172.21.15.84:0/1983269130 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f37a0050480 con 0x7f37b0142690 2024-03-29T10:38:41.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.946+0000 7f3797fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:41.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:41.946+0000 7f3797fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:42.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:42.272+0000 7f37b5a4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:42.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:42.272+0000 7f37b5a4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:42.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:42.272+0000 7f37b5a4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:38:42.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:42.272+0000 7f37b5a4a640 1 -- 172.21.15.84:0/1983269130 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37b012fec0 con 0x7f37b0142690 2024-03-29T10:38:42.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:42.273+0000 7f3797fff640 1 -- 172.21.15.84:0/1983269130 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f37a00523b0 con 0x7f37b0142690 2024-03-29T10:38:42.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:42.273+0000 7f3797fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:42.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:42.273+0000 7f3797fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:42.259 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:38:42.259 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:38:42.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:42.277+0000 7f37b5a4a640 10 librados: watch_flush enter 2024-03-29T10:38:42.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:42.278+0000 7f37b5a4a640 10 librados: watch_flush exit 2024-03-29T10:38:42.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:42.278+0000 7f37b5a4a640 20 client.4397.objecter shutdown clearing up homeless session... 2024-03-29T10:38:42.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:42.278+0000 7f37b5a4a640 10 client.4397.objecter successfully canceled tick 2024-03-29T10:38:42.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:42.279+0000 7f37b5a4a640 1 -- 172.21.15.84:0/1983269130 >> v1:172.21.15.86:6816/3337225337 conn(0x7f379c06ddb0 legacy=0x7f379c070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:42.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:42.279+0000 7f37b5a4a640 10 monclient: shutdown 2024-03-29T10:38:42.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:42.279+0000 7f37b5a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:42.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:42.279+0000 7f37b5a4a640 1 -- 172.21.15.84:0/1983269130 >> v1:172.21.15.84:6790/0 conn(0x7f37b0142690 legacy=0x7f37b00b0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:42.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:42.279+0000 7f37b5a4a640 1 -- 172.21.15.84:0/1983269130 shutdown_connections 2024-03-29T10:38:42.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:42.279+0000 7f37b5a4a640 1 -- 172.21.15.84:0/1983269130 >> 172.21.15.84:0/1983269130 conn(0x7f37b0066a00 msgr2=0x7f37b013dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:42.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:42.279+0000 7f37b5a4a640 1 -- 172.21.15.84:0/1983269130 shutdown_connections 2024-03-29T10:38:42.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:42.279+0000 7f37b5a4a640 1 -- 172.21.15.84:0/1983269130 wait complete. 2024-03-29T10:38:42.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:42.279+0000 7f37b5a4a640 1 librados: shutdown 2024-03-29T10:38:42.283 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:38:49.090 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:38:49.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.248+0000 7f061aede640 10 monclient: get_monmap_and_config 2024-03-29T10:38:49.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.248+0000 7f061aede640 10 monclient: build_initial_monmap 2024-03-29T10:38:49.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.248+0000 7f061aede640 10 monclient: monmap: 2024-03-29T10:38:49.233 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:38:49.233 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:49.233 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:38:49.249347+0000 2024-03-29T10:38:49.233 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:38:49.249347+0000 2024-03-29T10:38:49.233 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:49.233 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:49.233 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:38:49.233 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:38:49.233 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:38:49.233 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:49.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.249+0000 7f061aede640 1 Processor -- start 2024-03-29T10:38:49.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.249+0000 7f061aede640 1 -- start start 2024-03-29T10:38:49.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.249+0000 7f061aede640 10 monclient: init 2024-03-29T10:38:49.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.250+0000 7f061aede640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:49.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.250+0000 7f061aede640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:38:49.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.252+0000 7f061aede640 10 monclient(hunting): picked mon.noname-c con 0x7f06140b2790 addr v1:172.21.15.86:6789/0 2024-03-29T10:38:49.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.252+0000 7f061aede640 10 monclient(hunting): picked mon.noname-b con 0x7f0614143c80 addr v1:172.21.15.84:6790/0 2024-03-29T10:38:49.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.252+0000 7f061aede640 10 monclient(hunting): picked mon.noname-a con 0x7f0614146e30 addr v1:172.21.15.84:6789/0 2024-03-29T10:38:49.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.252+0000 7f061aede640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0614146290 con 0x7f0614146e30 2024-03-29T10:38:49.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.252+0000 7f061aede640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06140b3970 con 0x7f06140b2790 2024-03-29T10:38:49.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.252+0000 7f061aede640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f061406ee80 con 0x7f0614143c80 2024-03-29T10:38:49.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.252+0000 7f061aede640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:49.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.252+0000 7f061aede640 10 monclient(hunting): authenticate will time out at 1031.515732s 2024-03-29T10:38:49.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.252+0000 7f0619edc640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f06140b2790 0x7f0614143570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:34149/0 (socket says 172.21.15.84:34149) 2024-03-29T10:38:49.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.252+0000 7f0619edc640 1 -- 172.21.15.84:0/1837623171 learned_addr learned my addr 172.21.15.84:0/1837623171 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:38:49.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.253+0000 7f0618eda640 1 -- 172.21.15.84:0/1837623171 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3315384663 0 0) 0x7f061406ee80 con 0x7f0614143c80 2024-03-29T10:38:49.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.253+0000 7f0618eda640 10 monclient(hunting): _negotiate 2024-03-29T10:38:49.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.253+0000 7f0618eda640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:49.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.253+0000 7f0618eda640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:49.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.253+0000 7f0618eda640 10 monclient(hunting): my global_id is 4400 2024-03-29T10:38:49.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.253+0000 7f0618eda640 1 -- 172.21.15.84:0/1837623171 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0600003520 con 0x7f0614143c80 2024-03-29T10:38:49.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.253+0000 7f0618eda640 1 -- 172.21.15.84:0/1837623171 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2494791910 0 0) 0x7f0600003520 con 0x7f0614143c80 2024-03-29T10:38:49.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.253+0000 7f0618eda640 1 -- 172.21.15.84:0/1837623171 >> v1:172.21.15.86:6789/0 conn(0x7f06140b2790 legacy=0x7f0614143570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:49.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.253+0000 7f0618eda640 1 -- 172.21.15.84:0/1837623171 >> v1:172.21.15.84:6789/0 conn(0x7f0614146e30 legacy=0x7f0614151bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:38:49.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.256+0000 7f0618eda640 10 monclient: _finish_hunting 0 2024-03-29T10:38:49.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.256+0000 7f0618eda640 1 monclient: found mon.noname-b 2024-03-29T10:38:49.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.256+0000 7f0618eda640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:49.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.256+0000 7f0618eda640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:38:49.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.256+0000 7f0618eda640 1 -- 172.21.15.84:0/1837623171 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f061406f160 con 0x7f0614143c80 2024-03-29T10:38:49.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.256+0000 7f0618eda640 10 monclient: _finish_auth 0 2024-03-29T10:38:49.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.256+0000 7f0618eda640 10 monclient: _check_auth_tickets 2024-03-29T10:38:49.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.256+0000 7f0618eda640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:49.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.256+0000 7f0618eda640 1 -- 172.21.15.84:0/1837623171 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0604001690 con 0x7f0614143c80 2024-03-29T10:38:49.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.256+0000 7f0618eda640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:49.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.256+0000 7f0618eda640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:38:49.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.256+0000 7f0618eda640 10 monclient: dump: 2024-03-29T10:38:49.242 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:49.242 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:49.242 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:49.243 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:49.243 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:49.243 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:49.243 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:49.243 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:49.243 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:49.243 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:49.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.256+0000 7f061aede640 5 monclient: authenticate success, global_id 4400 2024-03-29T10:38:49.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.256+0000 7f061aede640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:49.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.261+0000 7f0618eda640 1 -- 172.21.15.84:0/1837623171 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06040018b0 con 0x7f0614143c80 2024-03-29T10:38:49.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.261+0000 7f0618eda640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:49.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.261+0000 7f0618eda640 1 -- 172.21.15.84:0/1837623171 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0604001cc0 con 0x7f0614143c80 2024-03-29T10:38:49.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.261+0000 7f0618eda640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:49.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.261+0000 7f0618eda640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:38:49.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.261+0000 7f0618eda640 10 monclient: dump: 2024-03-29T10:38:49.247 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:49.247 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:49.247 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:49.247 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:49.247 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:49.247 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:49.247 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:49.247 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:49.247 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:49.247 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:49.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.261+0000 7f061aede640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:49.247 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.261+0000 7f061aede640 4 set_mon_vals no callback set 2024-03-29T10:38:49.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.262+0000 7f061aede640 10 monclient: shutdown 2024-03-29T10:38:49.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.262+0000 7f061aede640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:49.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.262+0000 7f061aede640 1 -- 172.21.15.84:0/1837623171 >> v1:172.21.15.84:6790/0 conn(0x7f0614143c80 legacy=0x7f0614146720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:49.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.262+0000 7f061aede640 1 -- 172.21.15.84:0/1837623171 shutdown_connections 2024-03-29T10:38:49.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.262+0000 7f061aede640 1 -- 172.21.15.84:0/1837623171 >> 172.21.15.84:0/1837623171 conn(0x7f061406d070 msgr2=0x7f0614066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:49.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.263+0000 7f061aede640 1 -- 172.21.15.84:0/1837623171 shutdown_connections 2024-03-29T10:38:49.248 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.263+0000 7f061aede640 1 -- 172.21.15.84:0/1837623171 wait complete. 2024-03-29T10:38:49.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.265+0000 7f061aede640 10 monclient: build_initial_monmap 2024-03-29T10:38:49.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.265+0000 7f061aede640 10 monclient: monmap: 2024-03-29T10:38:49.249 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:38:49.249 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:49.249 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:38:49.250 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:38:49.250 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:49.250 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:49.250 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:38:49.250 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:38:49.250 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:38:49.250 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:49.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.265+0000 7f061aede640 1 librados: starting msgr at 2024-03-29T10:38:49.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.265+0000 7f061aede640 1 librados: starting objecter 2024-03-29T10:38:49.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.266+0000 7f061aede640 1 Processor -- start 2024-03-29T10:38:49.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.266+0000 7f061aede640 1 -- start start 2024-03-29T10:38:49.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.266+0000 7f061aede640 1 librados: setting wanted keys 2024-03-29T10:38:49.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.266+0000 7f061aede640 1 librados: calling monclient init 2024-03-29T10:38:49.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.266+0000 7f061aede640 10 monclient: init 2024-03-29T10:38:49.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.267+0000 7f061aede640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:49.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.267+0000 7f061aede640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:38:49.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.267+0000 7f061aede640 10 monclient(hunting): picked mon.noname-b con 0x7f06140b2790 addr v1:172.21.15.86:6789/0 2024-03-29T10:38:49.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.267+0000 7f06196db640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f0614143c80 0x7f06141d9ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:43870/0 (socket says 172.21.15.84:43870) 2024-03-29T10:38:49.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.267+0000 7f06196db640 1 -- 172.21.15.84:0/802395361 learned_addr learned my addr 172.21.15.84:0/802395361 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:38:49.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.267+0000 7f061aede640 10 monclient(hunting): picked mon.noname-c con 0x7f0614143c80 addr v1:172.21.15.84:6790/0 2024-03-29T10:38:49.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.268+0000 7f061aede640 10 monclient(hunting): picked mon.noname-a con 0x7f0614146e30 addr v1:172.21.15.84:6789/0 2024-03-29T10:38:49.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.268+0000 7f061aede640 1 -- 172.21.15.84:0/802395361 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06140b6ce0 con 0x7f0614146e30 2024-03-29T10:38:49.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.268+0000 7f061aede640 1 -- 172.21.15.84:0/802395361 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f061413a900 con 0x7f06140b2790 2024-03-29T10:38:49.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.268+0000 7f060a7fc640 10 client.?.objecter ms_handle_connect 0x7f0614143c80 2024-03-29T10:38:49.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.268+0000 7f060a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:49.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.271+0000 7f061aede640 1 -- 172.21.15.84:0/802395361 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f061413ab70 con 0x7f0614143c80 2024-03-29T10:38:49.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.271+0000 7f061aede640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:49.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.271+0000 7f061aede640 10 monclient(hunting): authenticate will time out at 1031.534277s 2024-03-29T10:38:49.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 10 client.?.objecter ms_handle_connect 0x7f0614146e30 2024-03-29T10:38:49.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:49.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 10 client.?.objecter ms_handle_connect 0x7f06140b2790 2024-03-29T10:38:49.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:49.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 1 -- 172.21.15.84:0/802395361 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 388382649 0 0) 0x7f061413ab70 con 0x7f0614143c80 2024-03-29T10:38:49.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:49.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:49.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:49.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 10 monclient(hunting): my global_id is 4403 2024-03-29T10:38:49.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 1 -- 172.21.15.84:0/802395361 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05f8004eb0 con 0x7f0614143c80 2024-03-29T10:38:49.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 1 -- 172.21.15.84:0/802395361 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 442646803 0 0) 0x7f06140b6ce0 con 0x7f0614146e30 2024-03-29T10:38:49.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:38:49.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:49.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:49.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 10 monclient(hunting): my global_id is 4380 2024-03-29T10:38:49.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 1 -- 172.21.15.84:0/802395361 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f061413ab70 con 0x7f0614146e30 2024-03-29T10:38:49.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 1 -- 172.21.15.84:0/802395361 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 608709396 0 0) 0x7f061413ab70 con 0x7f0614146e30 2024-03-29T10:38:49.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 1 -- 172.21.15.84:0/802395361 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06140b6ce0 con 0x7f0614146e30 2024-03-29T10:38:49.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 1 -- 172.21.15.84:0/802395361 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f060c001a40 con 0x7f0614146e30 2024-03-29T10:38:49.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:49.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:38:49.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 10 monclient(hunting): dump: 2024-03-29T10:38:49.259 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:49.259 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:49.260 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:49.260 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:49.260 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:49.260 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:49.260 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:49.260 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:49.260 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:49.260 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:49.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:49.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 1 -- 172.21.15.84:0/802395361 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4244740960 0 0) 0x7f05f8004eb0 con 0x7f0614143c80 2024-03-29T10:38:49.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 1 -- 172.21.15.84:0/802395361 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f061413ab70 con 0x7f0614143c80 2024-03-29T10:38:49.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 1 -- 172.21.15.84:0/802395361 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2445892412 0 0) 0x7f06140b6ce0 con 0x7f0614146e30 2024-03-29T10:38:49.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 1 -- 172.21.15.84:0/802395361 >> v1:172.21.15.84:6790/0 conn(0x7f0614143c80 legacy=0x7f06141d9ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:49.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 1 -- 172.21.15.84:0/802395361 >> v1:172.21.15.86:6789/0 conn(0x7f06140b2790 legacy=0x7f06141e3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:49.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:38:49.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 1 monclient: found mon.a 2024-03-29T10:38:49.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:49.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:38:49.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 1 -- 172.21.15.84:0/802395361 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06140b68b0 con 0x7f0614146e30 2024-03-29T10:38:49.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:38:49.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:38:49.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f060a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:49.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f061aede640 5 monclient: authenticate success, global_id 4380 2024-03-29T10:38:49.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f061aede640 10 monclient: _renew_subs 2024-03-29T10:38:49.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f061aede640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:38:49.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f061aede640 1 -- 172.21.15.84:0/802395361 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06141399a0 con 0x7f0614146e30 2024-03-29T10:38:49.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f061aede640 10 client.4380.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:49.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f061aede640 10 monclient: _renew_subs 2024-03-29T10:38:49.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f061aede640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:38:49.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f061aede640 1 -- 172.21.15.84:0/802395361 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f06141425a0 con 0x7f0614146e30 2024-03-29T10:38:49.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.273+0000 7f061aede640 1 librados: init done 2024-03-29T10:38:49.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.275+0000 7f061aede640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:49.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.275+0000 7f061aede640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:49.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.275+0000 7f061aede640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:38:49.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.275+0000 7f061aede640 1 -- 172.21.15.84:0/802395361 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0614139ca0 con 0x7f0614146e30 2024-03-29T10:38:49.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.276+0000 7f060a7fc640 1 -- 172.21.15.84:0/802395361 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f060c001ef0 con 0x7f0614146e30 2024-03-29T10:38:49.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.276+0000 7f060a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:49.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.276+0000 7f060a7fc640 1 -- 172.21.15.84:0/802395361 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f060c003b30 con 0x7f0614146e30 2024-03-29T10:38:49.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.276+0000 7f060a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:49.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.276+0000 7f060a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:49.265 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.276+0000 7f060a7fc640 10 monclient: dump: 2024-03-29T10:38:49.265 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:49.265 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:49.265 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:49.265 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:49.265 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:49.265 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:49.266 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:49.266 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:49.266 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:49.266 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:49.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.276+0000 7f060a7fc640 1 -- 172.21.15.84:0/802395361 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f060c003d50 con 0x7f0614146e30 2024-03-29T10:38:49.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.277+0000 7f060a7fc640 1 -- 172.21.15.84:0/802395361 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f060c0838a0 con 0x7f0614146e30 2024-03-29T10:38:49.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.277+0000 7f060a7fc640 10 client.4380.objecter ms_dispatch 0x7f0614000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:38:49.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.277+0000 7f060a7fc640 3 client.4380.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:38:49.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.277+0000 7f060a7fc640 3 client.4380.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:38:49.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.277+0000 7f060a7fc640 20 client.4380.objecter dump_active .. 0 homeless 2024-03-29T10:38:49.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.277+0000 7f060b7fe640 4 set_mon_vals no callback set 2024-03-29T10:38:49.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.279+0000 7f0619edc640 10 monclient: get_auth_request con 0x7f05f806df00 auth_method 0 2024-03-29T10:38:49.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.280+0000 7f060a7fc640 10 client.4380.objecter ms_handle_connect 0x7f05f806df00 2024-03-29T10:38:49.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.287+0000 7f060a7fc640 1 -- 172.21.15.84:0/802395361 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f060c04f340 con 0x7f0614146e30 2024-03-29T10:38:49.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.287+0000 7f060a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:49.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.287+0000 7f060a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:49.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.602+0000 7f061aede640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:49.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.602+0000 7f061aede640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:49.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.602+0000 7f061aede640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:38:49.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.602+0000 7f061aede640 1 -- 172.21.15.84:0/802395361 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06140b1e70 con 0x7f0614146e30 2024-03-29T10:38:49.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.605+0000 7f060a7fc640 1 -- 172.21.15.84:0/802395361 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f060c0528d0 con 0x7f0614146e30 2024-03-29T10:38:49.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.606+0000 7f060a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:49.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.606+0000 7f060a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:49.590 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:38:49.590 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:38:49.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.609+0000 7f061aede640 10 librados: watch_flush enter 2024-03-29T10:38:49.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.609+0000 7f061aede640 10 librados: watch_flush exit 2024-03-29T10:38:49.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.609+0000 7f061aede640 20 client.4380.objecter shutdown clearing up homeless session... 2024-03-29T10:38:49.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.609+0000 7f061aede640 10 client.4380.objecter successfully canceled tick 2024-03-29T10:38:49.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.609+0000 7f061aede640 1 -- 172.21.15.84:0/802395361 >> v1:172.21.15.86:6816/3337225337 conn(0x7f05f806df00 legacy=0x7f05f80703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:49.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.609+0000 7f061aede640 10 monclient: shutdown 2024-03-29T10:38:49.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.609+0000 7f061aede640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:49.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.609+0000 7f061aede640 1 -- 172.21.15.84:0/802395361 >> v1:172.21.15.84:6789/0 conn(0x7f0614146e30 legacy=0x7f06141ef6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:49.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.612+0000 7f061aede640 1 -- 172.21.15.84:0/802395361 shutdown_connections 2024-03-29T10:38:49.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.612+0000 7f061aede640 1 -- 172.21.15.84:0/802395361 >> 172.21.15.84:0/802395361 conn(0x7f061406d070 msgr2=0x7f0614069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:49.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.613+0000 7f061aede640 1 -- 172.21.15.84:0/802395361 shutdown_connections 2024-03-29T10:38:49.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.613+0000 7f061aede640 1 -- 172.21.15.84:0/802395361 wait complete. 2024-03-29T10:38:49.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:49.613+0000 7f061aede640 1 librados: shutdown 2024-03-29T10:38:49.613 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:38:56.419 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:38:56.564 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.579+0000 7f25226c1640 10 monclient: get_monmap_and_config 2024-03-29T10:38:56.564 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.579+0000 7f25226c1640 10 monclient: build_initial_monmap 2024-03-29T10:38:56.564 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.579+0000 7f25226c1640 10 monclient: monmap: 2024-03-29T10:38:56.564 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:38:56.564 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:56.564 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:38:56.580801+0000 2024-03-29T10:38:56.564 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:38:56.580801+0000 2024-03-29T10:38:56.565 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:56.565 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:56.565 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:38:56.565 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:38:56.565 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:38:56.565 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:56.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.580+0000 7f25226c1640 1 Processor -- start 2024-03-29T10:38:56.566 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.581+0000 7f25226c1640 1 -- start start 2024-03-29T10:38:56.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.581+0000 7f25226c1640 10 monclient: init 2024-03-29T10:38:56.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.581+0000 7f25226c1640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:56.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.581+0000 7f25226c1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:38:56.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.582+0000 7f25226c1640 10 monclient(hunting): picked mon.noname-c con 0x7f2514006800 addr v1:172.21.15.86:6789/0 2024-03-29T10:38:56.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.582+0000 7f25226c1640 10 monclient(hunting): picked mon.noname-a con 0x7f2514006be0 addr v1:172.21.15.84:6789/0 2024-03-29T10:38:56.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.582+0000 7f25226c1640 10 monclient(hunting): picked mon.noname-b con 0x7f2514031b60 addr v1:172.21.15.84:6790/0 2024-03-29T10:38:56.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.582+0000 7f25226c1640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2514029370 con 0x7f2514006be0 2024-03-29T10:38:56.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.582+0000 7f25226c1640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25140295e0 con 0x7f2514006800 2024-03-29T10:38:56.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.582+0000 7f25226c1640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2514031360 con 0x7f2514031b60 2024-03-29T10:38:56.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.582+0000 7f25226c1640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:56.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.582+0000 7f25226c1640 10 monclient(hunting): authenticate will time out at 1038.845215s 2024-03-29T10:38:56.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.588+0000 7f25216bf640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f2514006800 0x7f25140cad20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:55321/0 (socket says 172.21.15.84:55321) 2024-03-29T10:38:56.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.588+0000 7f25216bf640 1 -- 172.21.15.84:0/2504075881 learned_addr learned my addr 172.21.15.84:0/2504075881 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:38:56.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f2513fff640 1 -- 172.21.15.84:0/2504075881 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3581266449 0 0) 0x7f2514029370 con 0x7f2514006be0 2024-03-29T10:38:56.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f2513fff640 10 monclient(hunting): _negotiate 2024-03-29T10:38:56.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f2513fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:56.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f2513fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:56.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f2513fff640 10 monclient(hunting): my global_id is 4383 2024-03-29T10:38:56.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f2513fff640 1 -- 172.21.15.84:0/2504075881 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2508003520 con 0x7f2514006be0 2024-03-29T10:38:56.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f2513fff640 1 -- 172.21.15.84:0/2504075881 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3308468244 0 0) 0x7f2514031360 con 0x7f2514031b60 2024-03-29T10:38:56.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f2513fff640 10 monclient(hunting): _negotiate 2024-03-29T10:38:56.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f25216bf640 1 -- 172.21.15.84:0/2504075881 >> v1:172.21.15.86:6789/0 conn(0x7f2514006800 legacy=0x7f25140cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-29T10:38:56.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f25216bf640 1 -- 172.21.15.84:0/2504075881 >> v1:172.21.15.86:6789/0 conn(0x7f2514006800 legacy=0x7f25140cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:38:56.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f25216bf640 1 --1- 172.21.15.84:0/2504075881 >> v1:172.21.15.86:6789/0 conn(0x7f2514006800 0x7f25140cad20 :-1 s=OPENED pgs=486401 cs=1 l=1).handle_message read tag failed 2024-03-29T10:38:56.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f25216bf640 1 --1- 172.21.15.84:0/2504075881 >> v1:172.21.15.86:6789/0 conn(0x7f2514006800 0x7f25140cad20 :-1 s=OPENED pgs=486401 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:38:56.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f2513fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:56.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f2513fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:56.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f2513fff640 10 monclient(hunting): my global_id is 4406 2024-03-29T10:38:56.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f2513fff640 1 -- 172.21.15.84:0/2504075881 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2514029370 con 0x7f2514031b60 2024-03-29T10:38:56.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f2513fff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.86:6789/0 2024-03-29T10:38:56.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f2513fff640 1 -- 172.21.15.84:0/2504075881 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2702292879 0 0) 0x7f2508003520 con 0x7f2514006be0 2024-03-29T10:38:56.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f2513fff640 1 -- 172.21.15.84:0/2504075881 >> v1:172.21.15.84:6790/0 conn(0x7f2514031b60 legacy=0x7f25140d30c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:56.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f2513fff640 1 -- 172.21.15.84:0/2504075881 >> v1:172.21.15.86:6789/0 conn(0x7f2514006800 legacy=0x7f25140cad20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:38:56.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f2513fff640 10 monclient: _finish_hunting 0 2024-03-29T10:38:56.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f2513fff640 1 monclient: found mon.noname-a 2024-03-29T10:38:56.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f2513fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:56.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f2513fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T10:38:56.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f2513fff640 1 -- 172.21.15.84:0/2504075881 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2514031640 con 0x7f2514006be0 2024-03-29T10:38:56.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f2513fff640 10 monclient: _finish_auth 0 2024-03-29T10:38:56.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f2513fff640 10 monclient: _check_auth_tickets 2024-03-29T10:38:56.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f2513fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:56.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f2513fff640 1 -- 172.21.15.84:0/2504075881 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f250c001690 con 0x7f2514006be0 2024-03-29T10:38:56.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f2513fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:56.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f2513fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:38:56.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f2513fff640 10 monclient: dump: 2024-03-29T10:38:56.576 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:56.576 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:56.576 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:56.576 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:56.576 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:56.576 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:56.576 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:56.576 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:56.576 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:56.576 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:56.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f2513fff640 1 -- 172.21.15.84:0/2504075881 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f250c0018b0 con 0x7f2514006be0 2024-03-29T10:38:56.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.589+0000 7f25226c1640 5 monclient: authenticate success, global_id 4383 2024-03-29T10:38:56.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.590+0000 7f25226c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:38:56.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.590+0000 7f2513fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:56.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.590+0000 7f2513fff640 1 -- 172.21.15.84:0/2504075881 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f250c001bc0 con 0x7f2514006be0 2024-03-29T10:38:56.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.590+0000 7f25226c1640 10 monclient: get_monmap_and_config success 2024-03-29T10:38:56.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.590+0000 7f25226c1640 4 set_mon_vals no callback set 2024-03-29T10:38:56.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.590+0000 7f2513fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:56.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.590+0000 7f2513fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:38:56.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.590+0000 7f2513fff640 10 monclient: dump: 2024-03-29T10:38:56.577 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:56.577 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:56.577 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:56.577 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:56.577 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:56.577 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:56.577 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:56.578 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:56.578 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:56.578 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:56.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.591+0000 7f25226c1640 10 monclient: shutdown 2024-03-29T10:38:56.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.591+0000 7f25226c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:56.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.591+0000 7f25226c1640 1 -- 172.21.15.84:0/2504075881 >> v1:172.21.15.84:6789/0 conn(0x7f2514006be0 legacy=0x7f25140c4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:56.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.592+0000 7f25226c1640 1 -- 172.21.15.84:0/2504075881 shutdown_connections 2024-03-29T10:38:56.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.592+0000 7f25226c1640 1 -- 172.21.15.84:0/2504075881 >> 172.21.15.84:0/2504075881 conn(0x7f25140058f0 msgr2=0x7f2514005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:56.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.592+0000 7f25226c1640 1 -- 172.21.15.84:0/2504075881 shutdown_connections 2024-03-29T10:38:56.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.592+0000 7f25226c1640 1 -- 172.21.15.84:0/2504075881 wait complete. 2024-03-29T10:38:56.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.594+0000 7f25226c1640 10 monclient: build_initial_monmap 2024-03-29T10:38:56.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.594+0000 7f25226c1640 10 monclient: monmap: 2024-03-29T10:38:56.580 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:38:56.580 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:38:56.580 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:38:56.580 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:38:56.580 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:38:56.580 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:56.580 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:38:56.580 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:38:56.581 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:38:56.581 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:56.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.594+0000 7f25226c1640 1 librados: starting msgr at 2024-03-29T10:38:56.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.594+0000 7f25226c1640 1 librados: starting objecter 2024-03-29T10:38:56.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.594+0000 7f25226c1640 1 Processor -- start 2024-03-29T10:38:56.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.594+0000 7f25226c1640 1 -- start start 2024-03-29T10:38:56.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.594+0000 7f25226c1640 1 librados: setting wanted keys 2024-03-29T10:38:56.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.594+0000 7f25226c1640 1 librados: calling monclient init 2024-03-29T10:38:56.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.594+0000 7f25226c1640 10 monclient: init 2024-03-29T10:38:56.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.595+0000 7f25226c1640 10 monclient: _reopen_session rank -1 2024-03-29T10:38:56.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.595+0000 7f25226c1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:38:56.585 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.596+0000 7f25226c1640 10 monclient(hunting): picked mon.noname-c con 0x7f2514006800 addr v1:172.21.15.84:6790/0 2024-03-29T10:38:56.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.596+0000 7f25226c1640 10 monclient(hunting): picked mon.noname-a con 0x7f2514031b60 addr v1:172.21.15.84:6789/0 2024-03-29T10:38:56.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.597+0000 7f25226c1640 10 monclient(hunting): picked mon.noname-b con 0x7f25140c63f0 addr v1:172.21.15.86:6789/0 2024-03-29T10:38:56.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.597+0000 7f25226c1640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25140c6810 con 0x7f2514031b60 2024-03-29T10:38:56.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.597+0000 7f25226c1640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25140c0330 con 0x7f25140c63f0 2024-03-29T10:38:56.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.597+0000 7f25226c1640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25140c7730 con 0x7f2514006800 2024-03-29T10:38:56.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.597+0000 7f25226c1640 10 monclient(hunting): _renew_subs 2024-03-29T10:38:56.586 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.597+0000 7f25226c1640 10 monclient(hunting): authenticate will time out at 1038.860914s 2024-03-29T10:38:56.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.601+0000 7f2520ebe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f2514031b60 0x7f25140bf510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:39627/0 (socket says 172.21.15.84:39627) 2024-03-29T10:38:56.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.601+0000 7f2520ebe640 1 -- 172.21.15.84:0/417063617 learned_addr learned my addr 172.21.15.84:0/417063617 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:38:56.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.602+0000 7f2511ffb640 10 client.?.objecter ms_handle_connect 0x7f2514031b60 2024-03-29T10:38:56.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.602+0000 7f2511ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:56.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.602+0000 7f2511ffb640 1 -- 172.21.15.84:0/417063617 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3386385750 0 0) 0x7f25140c6810 con 0x7f2514031b60 2024-03-29T10:38:56.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.602+0000 7f2511ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:38:56.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.602+0000 7f2511ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:56.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.602+0000 7f2511ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:56.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.602+0000 7f2511ffb640 10 monclient(hunting): my global_id is 4386 2024-03-29T10:38:56.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.602+0000 7f2511ffb640 1 -- 172.21.15.84:0/417063617 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2500004eb0 con 0x7f2514031b60 2024-03-29T10:38:56.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.603+0000 7f2511ffb640 10 client.?.objecter ms_handle_connect 0x7f25140c63f0 2024-03-29T10:38:56.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.603+0000 7f2511ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:38:56.588 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.603+0000 7f2511ffb640 1 -- 172.21.15.84:0/417063617 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 251612032 0 0) 0x7f2500004eb0 con 0x7f2514031b60 2024-03-29T10:38:56.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.603+0000 7f2511ffb640 1 -- 172.21.15.84:0/417063617 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25140c6810 con 0x7f2514031b60 2024-03-29T10:38:56.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.603+0000 7f2511ffb640 1 -- 172.21.15.84:0/417063617 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f250c001610 con 0x7f2514031b60 2024-03-29T10:38:56.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.603+0000 7f2511ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:56.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.603+0000 7f2511ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:38:56.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.603+0000 7f2511ffb640 10 monclient(hunting): dump: 2024-03-29T10:38:56.589 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:56.589 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:56.589 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:56.589 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:56.589 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:56.589 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:56.589 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:56.589 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:56.590 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:56.590 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:56.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.603+0000 7f2511ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:38:56.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.603+0000 7f2511ffb640 1 -- 172.21.15.84:0/417063617 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4034976166 0 0) 0x7f25140c0330 con 0x7f25140c63f0 2024-03-29T10:38:56.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.603+0000 7f2511ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:38:56.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.603+0000 7f2511ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:38:56.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.603+0000 7f2511ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:38:56.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.603+0000 7f2511ffb640 10 monclient(hunting): my global_id is 4360 2024-03-29T10:38:56.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.603+0000 7f2511ffb640 1 -- 172.21.15.84:0/417063617 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2500004eb0 con 0x7f25140c63f0 2024-03-29T10:38:56.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.603+0000 7f2511ffb640 1 -- 172.21.15.84:0/417063617 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1197675633 0 0) 0x7f2500004eb0 con 0x7f25140c63f0 2024-03-29T10:38:56.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.603+0000 7f2511ffb640 1 -- 172.21.15.84:0/417063617 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f25140c0330 con 0x7f25140c63f0 2024-03-29T10:38:56.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.604+0000 7f2511ffb640 1 -- 172.21.15.84:0/417063617 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f251c04fdd0 con 0x7f25140c63f0 2024-03-29T10:38:56.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.604+0000 7f2511ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:56.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.604+0000 7f2511ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:56.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.604+0000 7f2511ffb640 10 monclient(hunting): dump: 2024-03-29T10:38:56.593 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:56.593 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:56.593 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:56.593 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:56.593 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:56.593 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:56.593 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:56.593 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:56.593 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:56.593 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:56.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.604+0000 7f2511ffb640 1 -- 172.21.15.84:0/417063617 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2035288632 0 0) 0x7f25140c0330 con 0x7f25140c63f0 2024-03-29T10:38:56.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.604+0000 7f2511ffb640 1 -- 172.21.15.84:0/417063617 >> v1:172.21.15.84:6790/0 conn(0x7f2514006800 legacy=0x7f25140c7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:56.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.604+0000 7f2511ffb640 1 -- 172.21.15.84:0/417063617 >> v1:172.21.15.84:6789/0 conn(0x7f2514031b60 legacy=0x7f25140bf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:56.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.604+0000 7f2511ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:38:56.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.604+0000 7f2511ffb640 1 monclient: found mon.b 2024-03-29T10:38:56.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.604+0000 7f2511ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:38:56.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.604+0000 7f2511ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:38:56.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.604+0000 7f2511ffb640 1 -- 172.21.15.84:0/417063617 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25140c7a10 con 0x7f25140c63f0 2024-03-29T10:38:56.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.604+0000 7f2511ffb640 10 monclient: _finish_auth 0 2024-03-29T10:38:56.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.604+0000 7f2511ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:38:56.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.604+0000 7f2511ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:38:56.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.604+0000 7f2511ffb640 10 client.?.objecter ms_handle_connect 0x7f2514006800 2024-03-29T10:38:56.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.604+0000 7f25226c1640 5 monclient: authenticate success, global_id 4360 2024-03-29T10:38:56.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.604+0000 7f25226c1640 10 monclient: _renew_subs 2024-03-29T10:38:56.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.604+0000 7f25226c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:38:56.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.604+0000 7f25226c1640 1 -- 172.21.15.84:0/417063617 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2514030600 con 0x7f25140c63f0 2024-03-29T10:38:56.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.604+0000 7f2511ffb640 10 client.4360.objecter resend_mon_ops 2024-03-29T10:38:56.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.605+0000 7f2511ffb640 1 -- 172.21.15.84:0/417063617 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f251c04f400 con 0x7f25140c63f0 2024-03-29T10:38:56.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.605+0000 7f2511ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:38:56.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.605+0000 7f25226c1640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:38:56.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.605+0000 7f2511ffb640 1 -- 172.21.15.84:0/417063617 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f251c075710 con 0x7f25140c63f0 2024-03-29T10:38:56.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.605+0000 7f25137fe640 4 set_mon_vals no callback set 2024-03-29T10:38:56.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.605+0000 7f25226c1640 10 monclient: _renew_subs 2024-03-29T10:38:56.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.605+0000 7f25226c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:38:56.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.605+0000 7f25226c1640 1 -- 172.21.15.84:0/417063617 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f251402f200 con 0x7f25140c63f0 2024-03-29T10:38:56.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.605+0000 7f25226c1640 1 librados: init done 2024-03-29T10:38:56.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.606+0000 7f2511ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:38:56.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.606+0000 7f2511ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:38:56.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.606+0000 7f2511ffb640 10 monclient: dump: 2024-03-29T10:38:56.595 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:38:56.595 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:38:56.595 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:56.596 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:38:56.596 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:38:56.596 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:38:56.596 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:38:56.596 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:38:56.596 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:38:56.596 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:38:56.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.606+0000 7f25077fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:56.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.606+0000 7f25077fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:56.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.606+0000 7f25077fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:38:56.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.606+0000 7f25077fe640 1 -- 172.21.15.84:0/417063617 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2514004cd0 con 0x7f25140c63f0 2024-03-29T10:38:56.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.607+0000 7f2511ffb640 1 -- 172.21.15.84:0/417063617 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f251c067bf0 con 0x7f25140c63f0 2024-03-29T10:38:56.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.608+0000 7f2511ffb640 1 -- 172.21.15.84:0/417063617 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f251c0fe0d0 con 0x7f25140c63f0 2024-03-29T10:38:56.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.608+0000 7f2511ffb640 10 client.4360.objecter ms_dispatch 0x7f251c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:38:56.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.608+0000 7f2511ffb640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:38:56.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.608+0000 7f2511ffb640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:38:56.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.608+0000 7f2511ffb640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-29T10:38:56.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.609+0000 7f25216bf640 10 monclient: get_auth_request con 0x7f250006dd70 auth_method 0 2024-03-29T10:38:56.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.611+0000 7f2511ffb640 10 client.4360.objecter ms_handle_connect 0x7f250006dd70 2024-03-29T10:38:56.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.616+0000 7f2511ffb640 1 -- 172.21.15.84:0/417063617 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f251c0c99b0 con 0x7f25140c63f0 2024-03-29T10:38:56.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.616+0000 7f2511ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:38:56.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.616+0000 7f2511ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:38:56.920 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.935+0000 7f25077fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:56.920 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.935+0000 7f25077fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:56.920 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.935+0000 7f25077fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:38:56.920 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.935+0000 7f25077fe640 1 -- 172.21.15.84:0/417063617 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25140008d0 con 0x7f25140c63f0 2024-03-29T10:38:56.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.940+0000 7f2511ffb640 1 -- 172.21.15.84:0/417063617 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f251c0cb8e0 con 0x7f25140c63f0 2024-03-29T10:38:56.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.940+0000 7f2511ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:38:56.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.940+0000 7f2511ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:38:56.924 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:38:56.924 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:38:56.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.943+0000 7f25077fe640 10 librados: watch_flush enter 2024-03-29T10:38:56.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.943+0000 7f25077fe640 10 librados: watch_flush exit 2024-03-29T10:38:56.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.943+0000 7f25077fe640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-29T10:38:56.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.944+0000 7f25077fe640 10 client.4360.objecter successfully canceled tick 2024-03-29T10:38:56.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.944+0000 7f25077fe640 1 -- 172.21.15.84:0/417063617 >> v1:172.21.15.86:6816/3337225337 conn(0x7f250006dd70 legacy=0x7f2500070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:56.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.944+0000 7f25077fe640 10 monclient: shutdown 2024-03-29T10:38:56.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.944+0000 7f25077fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:38:56.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.944+0000 7f25077fe640 1 -- 172.21.15.84:0/417063617 >> v1:172.21.15.86:6789/0 conn(0x7f25140c63f0 legacy=0x7f25140bfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:38:56.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.944+0000 7f25077fe640 1 -- 172.21.15.84:0/417063617 shutdown_connections 2024-03-29T10:38:56.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.944+0000 7f25077fe640 1 -- 172.21.15.84:0/417063617 >> 172.21.15.84:0/417063617 conn(0x7f25140058f0 msgr2=0x7f2514036f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:38:56.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.944+0000 7f25077fe640 1 -- 172.21.15.84:0/417063617 shutdown_connections 2024-03-29T10:38:56.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.944+0000 7f25077fe640 1 -- 172.21.15.84:0/417063617 wait complete. 2024-03-29T10:38:56.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:38:56.944+0000 7f25077fe640 1 librados: shutdown 2024-03-29T10:38:56.947 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:03.755 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:03.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.919+0000 7f1737246640 10 monclient: get_monmap_and_config 2024-03-29T10:39:03.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.919+0000 7f1737246640 10 monclient: build_initial_monmap 2024-03-29T10:39:03.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.919+0000 7f1737246640 10 monclient: monmap: 2024-03-29T10:39:03.905 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:39:03.905 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:03.905 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:39:03.920399+0000 2024-03-29T10:39:03.905 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:39:03.920399+0000 2024-03-29T10:39:03.905 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:03.905 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:03.905 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:39:03.905 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:39:03.905 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:39:03.905 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:03.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.921+0000 7f1737246640 1 Processor -- start 2024-03-29T10:39:03.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.921+0000 7f1737246640 1 -- start start 2024-03-29T10:39:03.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.921+0000 7f1737246640 10 monclient: init 2024-03-29T10:39:03.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.922+0000 7f1737246640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:03.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.922+0000 7f1737246640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:39:03.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.922+0000 7f1737246640 10 monclient(hunting): picked mon.noname-b con 0x7f1728006410 addr v1:172.21.15.84:6790/0 2024-03-29T10:39:03.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.922+0000 7f1737246640 10 monclient(hunting): picked mon.noname-c con 0x7f172802a640 addr v1:172.21.15.86:6789/0 2024-03-29T10:39:03.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.922+0000 7f1737246640 10 monclient(hunting): picked mon.noname-a con 0x7f172802b370 addr v1:172.21.15.84:6789/0 2024-03-29T10:39:03.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.922+0000 7f1737246640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f172802a120 con 0x7f172802b370 2024-03-29T10:39:03.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.922+0000 7f1737246640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17280c75b0 con 0x7f172802a640 2024-03-29T10:39:03.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.922+0000 7f1737246640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1728033110 con 0x7f1728006410 2024-03-29T10:39:03.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.922+0000 7f1737246640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:03.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.922+0000 7f1737246640 10 monclient(hunting): authenticate will time out at 1046.185912s 2024-03-29T10:39:03.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.926+0000 7f1736a45640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f172802b370 0x7f17280320c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:44403/0 (socket says 172.21.15.84:44403) 2024-03-29T10:39:03.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.926+0000 7f1736a45640 1 -- 172.21.15.84:0/2634859946 learned_addr learned my addr 172.21.15.84:0/2634859946 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:39:03.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.927+0000 7f1735242640 1 -- 172.21.15.84:0/2634859946 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2568536715 0 0) 0x7f172802a120 con 0x7f172802b370 2024-03-29T10:39:03.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.927+0000 7f1735242640 10 monclient(hunting): _negotiate 2024-03-29T10:39:03.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.927+0000 7f1735242640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:03.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.927+0000 7f1735242640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:03.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.927+0000 7f1735242640 10 monclient(hunting): my global_id is 4389 2024-03-29T10:39:03.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.927+0000 7f1735242640 1 -- 172.21.15.84:0/2634859946 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f171c003520 con 0x7f172802b370 2024-03-29T10:39:03.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.927+0000 7f1735242640 1 -- 172.21.15.84:0/2634859946 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3746294957 0 0) 0x7f171c003520 con 0x7f172802b370 2024-03-29T10:39:03.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.927+0000 7f1735242640 1 -- 172.21.15.84:0/2634859946 >> v1:172.21.15.84:6790/0 conn(0x7f1728006410 legacy=0x7f17280345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:39:03.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.927+0000 7f1735242640 1 -- 172.21.15.84:0/2634859946 >> v1:172.21.15.86:6789/0 conn(0x7f172802a640 legacy=0x7f17280c6ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:39:03.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.928+0000 7f1735242640 10 monclient: _finish_hunting 0 2024-03-29T10:39:03.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.928+0000 7f1735242640 1 monclient: found mon.noname-a 2024-03-29T10:39:03.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.928+0000 7f1735242640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:03.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.928+0000 7f1735242640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T10:39:03.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.928+0000 7f1735242640 1 -- 172.21.15.84:0/2634859946 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17280332a0 con 0x7f172802b370 2024-03-29T10:39:03.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.928+0000 7f1735242640 10 monclient: _finish_auth 0 2024-03-29T10:39:03.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.928+0000 7f1735242640 10 monclient: _check_auth_tickets 2024-03-29T10:39:03.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.928+0000 7f1735242640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:03.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.928+0000 7f1735242640 1 -- 172.21.15.84:0/2634859946 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f172c001690 con 0x7f172802b370 2024-03-29T10:39:03.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.928+0000 7f1735242640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:03.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.928+0000 7f1735242640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:39:03.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.928+0000 7f1735242640 10 monclient: dump: 2024-03-29T10:39:03.914 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:03.914 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:03.914 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:03.914 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:03.914 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:03.914 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:03.914 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:03.914 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:03.914 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:03.914 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:03.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.928+0000 7f1737246640 5 monclient: authenticate success, global_id 4389 2024-03-29T10:39:03.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.928+0000 7f1737246640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:03.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.932+0000 7f1735242640 1 -- 172.21.15.84:0/2634859946 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f172c0017f0 con 0x7f172802b370 2024-03-29T10:39:03.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.932+0000 7f1735242640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:03.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.932+0000 7f1735242640 1 -- 172.21.15.84:0/2634859946 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f172c001c40 con 0x7f172802b370 2024-03-29T10:39:03.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.932+0000 7f1735242640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:03.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.932+0000 7f1735242640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:03.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.932+0000 7f1735242640 10 monclient: dump: 2024-03-29T10:39:03.917 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:03.917 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:03.917 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:03.917 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:03.917 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:03.917 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:03.917 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:03.917 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:03.917 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:03.917 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:03.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.932+0000 7f1737246640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:03.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.932+0000 7f1737246640 4 set_mon_vals no callback set 2024-03-29T10:39:03.918 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.933+0000 7f1737246640 10 monclient: shutdown 2024-03-29T10:39:03.918 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.933+0000 7f1737246640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:03.918 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.933+0000 7f1737246640 1 -- 172.21.15.84:0/2634859946 >> v1:172.21.15.84:6789/0 conn(0x7f172802b370 legacy=0x7f17280320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:03.920 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.936+0000 7f1737246640 1 -- 172.21.15.84:0/2634859946 shutdown_connections 2024-03-29T10:39:03.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.936+0000 7f1737246640 1 -- 172.21.15.84:0/2634859946 >> 172.21.15.84:0/2634859946 conn(0x7f17280058f0 msgr2=0x7f1728005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:03.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.936+0000 7f1737246640 1 -- 172.21.15.84:0/2634859946 shutdown_connections 2024-03-29T10:39:03.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.936+0000 7f1737246640 1 -- 172.21.15.84:0/2634859946 wait complete. 2024-03-29T10:39:03.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.937+0000 7f1737246640 10 monclient: build_initial_monmap 2024-03-29T10:39:03.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.937+0000 7f1737246640 10 monclient: monmap: 2024-03-29T10:39:03.921 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:39:03.921 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:03.922 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:39:03.922 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:39:03.922 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:03.922 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:03.922 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:39:03.922 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:39:03.922 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:39:03.922 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:03.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.937+0000 7f1737246640 1 librados: starting msgr at 2024-03-29T10:39:03.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.937+0000 7f1737246640 1 librados: starting objecter 2024-03-29T10:39:03.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.938+0000 7f1737246640 1 Processor -- start 2024-03-29T10:39:03.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.938+0000 7f1737246640 1 -- start start 2024-03-29T10:39:03.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.938+0000 7f1737246640 1 librados: setting wanted keys 2024-03-29T10:39:03.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.938+0000 7f1737246640 1 librados: calling monclient init 2024-03-29T10:39:03.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.938+0000 7f1737246640 10 monclient: init 2024-03-29T10:39:03.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.939+0000 7f1737246640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:03.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.939+0000 7f1737246640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:39:03.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.939+0000 7f1737246640 10 monclient(hunting): picked mon.noname-a con 0x7f1728006410 addr v1:172.21.15.84:6789/0 2024-03-29T10:39:03.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.939+0000 7f1737246640 10 monclient(hunting): picked mon.noname-c con 0x7f172802a640 addr v1:172.21.15.84:6790/0 2024-03-29T10:39:03.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.939+0000 7f1737246640 10 monclient(hunting): picked mon.noname-b con 0x7f172802b370 addr v1:172.21.15.86:6789/0 2024-03-29T10:39:03.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.939+0000 7f1737246640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f172802e8c0 con 0x7f1728006410 2024-03-29T10:39:03.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.939+0000 7f1737246640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f172802daf0 con 0x7f172802b370 2024-03-29T10:39:03.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.939+0000 7f1737246640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17280303d0 con 0x7f172802a640 2024-03-29T10:39:03.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.939+0000 7f1737246640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:03.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.939+0000 7f1737246640 10 monclient(hunting): authenticate will time out at 1046.202409s 2024-03-29T10:39:03.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.944+0000 7f1736a45640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f172802b370 0x7f17280c3560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:44981/0 (socket says 172.21.15.84:44981) 2024-03-29T10:39:03.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.944+0000 7f1736a45640 1 -- 172.21.15.84:0/315423686 learned_addr learned my addr 172.21.15.84:0/315423686 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:39:03.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.946+0000 7f1726ffd640 10 client.?.objecter ms_handle_connect 0x7f172802b370 2024-03-29T10:39:03.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.946+0000 7f1726ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:03.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.946+0000 7f1726ffd640 1 -- 172.21.15.84:0/315423686 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3419147726 0 0) 0x7f172802daf0 con 0x7f172802b370 2024-03-29T10:39:03.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.946+0000 7f1726ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:39:03.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.946+0000 7f1726ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:03.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.946+0000 7f1726ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:03.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.946+0000 7f1726ffd640 10 monclient(hunting): my global_id is 4363 2024-03-29T10:39:03.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.946+0000 7f1726ffd640 1 -- 172.21.15.84:0/315423686 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1714004eb0 con 0x7f172802b370 2024-03-29T10:39:03.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.947+0000 7f1726ffd640 1 -- 172.21.15.84:0/315423686 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4079201553 0 0) 0x7f1714004eb0 con 0x7f172802b370 2024-03-29T10:39:03.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.947+0000 7f1726ffd640 1 -- 172.21.15.84:0/315423686 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f172802daf0 con 0x7f172802b370 2024-03-29T10:39:03.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.947+0000 7f1726ffd640 1 -- 172.21.15.84:0/315423686 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f172c001b00 con 0x7f172802b370 2024-03-29T10:39:03.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.947+0000 7f1726ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:03.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.947+0000 7f1726ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:03.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.947+0000 7f1726ffd640 10 monclient(hunting): dump: 2024-03-29T10:39:03.932 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:03.932 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:03.932 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:03.933 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:03.933 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:03.933 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:03.933 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:03.933 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:03.933 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:03.933 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:03.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.947+0000 7f1726ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:03.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.947+0000 7f1726ffd640 1 -- 172.21.15.84:0/315423686 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1984328601 0 0) 0x7f172802daf0 con 0x7f172802b370 2024-03-29T10:39:03.933 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.947+0000 7f1726ffd640 1 -- 172.21.15.84:0/315423686 >> v1:172.21.15.84:6790/0 conn(0x7f172802a640 legacy=0x7f172802cc40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:39:03.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.949+0000 7f1726ffd640 1 -- 172.21.15.84:0/315423686 >> v1:172.21.15.84:6789/0 conn(0x7f1728006410 legacy=0x7f172802ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:03.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.949+0000 7f1726ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:39:03.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.949+0000 7f1726ffd640 1 monclient: found mon.b 2024-03-29T10:39:03.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.949+0000 7f1726ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:03.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.949+0000 7f1726ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:39:03.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.949+0000 7f1726ffd640 1 -- 172.21.15.84:0/315423686 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17280bd690 con 0x7f172802b370 2024-03-29T10:39:03.936 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.949+0000 7f1726ffd640 10 monclient: _finish_auth 0 2024-03-29T10:39:03.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.949+0000 7f1726ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:39:03.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.949+0000 7f1726ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:03.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.949+0000 7f1726ffd640 10 client.?.objecter ms_handle_connect 0x7f1728006410 2024-03-29T10:39:03.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.949+0000 7f1726ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:03.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.949+0000 7f1737246640 5 monclient: authenticate success, global_id 4363 2024-03-29T10:39:03.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.949+0000 7f1737246640 10 monclient: _renew_subs 2024-03-29T10:39:03.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.949+0000 7f1737246640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:39:03.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.949+0000 7f1737246640 1 -- 172.21.15.84:0/315423686 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17280bd8e0 con 0x7f172802b370 2024-03-29T10:39:03.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.949+0000 7f1737246640 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:03.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.949+0000 7f1737246640 10 monclient: _renew_subs 2024-03-29T10:39:03.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.949+0000 7f1737246640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:39:03.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.949+0000 7f1737246640 1 -- 172.21.15.84:0/315423686 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f172802df70 con 0x7f172802b370 2024-03-29T10:39:03.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.949+0000 7f1737246640 1 librados: init done 2024-03-29T10:39:03.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.950+0000 7f1726ffd640 1 -- 172.21.15.84:0/315423686 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f172c004bf0 con 0x7f172802b370 2024-03-29T10:39:03.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.950+0000 7f1726ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:03.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.950+0000 7f1734a41640 4 set_mon_vals no callback set 2024-03-29T10:39:03.937 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.951+0000 7f1737246640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:03.938 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.951+0000 7f1726ffd640 1 -- 172.21.15.84:0/315423686 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f172c001b00 con 0x7f172802b370 2024-03-29T10:39:03.938 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.952+0000 7f1737246640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:03.939 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.953+0000 7f1737246640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:39:03.939 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.953+0000 7f1737246640 1 -- 172.21.15.84:0/315423686 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1718005080 con 0x7f172802b370 2024-03-29T10:39:03.939 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.953+0000 7f1726ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:03.939 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.953+0000 7f1726ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:03.939 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.953+0000 7f1726ffd640 10 monclient: dump: 2024-03-29T10:39:03.939 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:03.940 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:03.940 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:03.940 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:03.940 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:03.940 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:03.940 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:03.940 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:03.940 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:03.940 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:03.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.953+0000 7f1726ffd640 1 -- 172.21.15.84:0/315423686 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f172c01abd0 con 0x7f172802b370 2024-03-29T10:39:03.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.954+0000 7f1726ffd640 1 -- 172.21.15.84:0/315423686 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f172c084a00 con 0x7f172802b370 2024-03-29T10:39:03.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.954+0000 7f1726ffd640 10 client.4363.objecter ms_dispatch 0x7f1730000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:03.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.954+0000 7f1726ffd640 3 client.4363.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:03.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.954+0000 7f1726ffd640 3 client.4363.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:03.940 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.954+0000 7f1726ffd640 20 client.4363.objecter dump_active .. 0 homeless 2024-03-29T10:39:03.941 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.954+0000 7f1736244640 10 monclient: get_auth_request con 0x7f171406de40 auth_method 0 2024-03-29T10:39:03.941 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.955+0000 7f1726ffd640 10 client.4363.objecter ms_handle_connect 0x7f171406de40 2024-03-29T10:39:03.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.960+0000 7f1726ffd640 1 -- 172.21.15.84:0/315423686 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f172c0502e0 con 0x7f172802b370 2024-03-29T10:39:03.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.960+0000 7f1726ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:03.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:03.960+0000 7f1726ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:04.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:04.284+0000 7f1737246640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:04.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:04.284+0000 7f1737246640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:04.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:04.284+0000 7f1737246640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:39:04.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:04.284+0000 7f1737246640 1 -- 172.21.15.84:0/315423686 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1718005330 con 0x7f172802b370 2024-03-29T10:39:04.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:04.286+0000 7f1726ffd640 1 -- 172.21.15.84:0/315423686 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f172c0502e0 con 0x7f172802b370 2024-03-29T10:39:04.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:04.286+0000 7f1726ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:04.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:04.286+0000 7f1726ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:04.271 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:39:04.271 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:04.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:04.289+0000 7f1737246640 10 librados: watch_flush enter 2024-03-29T10:39:04.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:04.290+0000 7f1737246640 10 librados: watch_flush exit 2024-03-29T10:39:04.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:04.290+0000 7f1737246640 20 client.4363.objecter shutdown clearing up homeless session... 2024-03-29T10:39:04.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:04.290+0000 7f1737246640 10 client.4363.objecter successfully canceled tick 2024-03-29T10:39:04.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:04.290+0000 7f1737246640 1 -- 172.21.15.84:0/315423686 >> v1:172.21.15.86:6816/3337225337 conn(0x7f171406de40 legacy=0x7f1714070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:04.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:04.290+0000 7f1737246640 10 monclient: shutdown 2024-03-29T10:39:04.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:04.290+0000 7f1737246640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:04.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:04.290+0000 7f1737246640 1 -- 172.21.15.84:0/315423686 >> v1:172.21.15.86:6789/0 conn(0x7f172802b370 legacy=0x7f17280c3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:04.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:04.290+0000 7f1737246640 1 -- 172.21.15.84:0/315423686 shutdown_connections 2024-03-29T10:39:04.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:04.290+0000 7f1737246640 1 -- 172.21.15.84:0/315423686 >> 172.21.15.84:0/315423686 conn(0x7f17280058f0 msgr2=0x7f17280043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:04.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:04.290+0000 7f1737246640 1 -- 172.21.15.84:0/315423686 shutdown_connections 2024-03-29T10:39:04.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:04.290+0000 7f1737246640 1 -- 172.21.15.84:0/315423686 wait complete. 2024-03-29T10:39:04.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:04.290+0000 7f1737246640 1 librados: shutdown 2024-03-29T10:39:04.290 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:11.097 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:11.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.272+0000 7f5a1a092640 10 monclient: get_monmap_and_config 2024-03-29T10:39:11.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.272+0000 7f5a1a092640 10 monclient: build_initial_monmap 2024-03-29T10:39:11.257 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.272+0000 7f5a1a092640 10 monclient: monmap: 2024-03-29T10:39:11.257 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:39:11.257 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:11.257 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:39:11.273134+0000 2024-03-29T10:39:11.257 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:39:11.273134+0000 2024-03-29T10:39:11.257 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:11.257 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:11.257 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:39:11.257 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:39:11.258 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:39:11.258 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:11.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.275+0000 7f5a1a092640 1 Processor -- start 2024-03-29T10:39:11.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.276+0000 7f5a1a092640 1 -- start start 2024-03-29T10:39:11.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.276+0000 7f5a1a092640 10 monclient: init 2024-03-29T10:39:11.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.277+0000 7f5a1a092640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:11.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.277+0000 7f5a1a092640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:39:11.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.277+0000 7f5a1a092640 10 monclient(hunting): picked mon.noname-b con 0x7f5a140a9750 addr v1:172.21.15.84:6790/0 2024-03-29T10:39:11.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.277+0000 7f5a1a092640 10 monclient(hunting): picked mon.noname-c con 0x7f5a140adca0 addr v1:172.21.15.86:6789/0 2024-03-29T10:39:11.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.277+0000 7f5a1a092640 10 monclient(hunting): picked mon.noname-a con 0x7f5a14146a80 addr v1:172.21.15.84:6789/0 2024-03-29T10:39:11.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.277+0000 7f5a1a092640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a140abdc0 con 0x7f5a14146a80 2024-03-29T10:39:11.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.277+0000 7f5a1a092640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a140b0070 con 0x7f5a140adca0 2024-03-29T10:39:11.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.277+0000 7f5a1a092640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a140af880 con 0x7f5a140a9750 2024-03-29T10:39:11.262 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.277+0000 7f5a1a092640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:11.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.277+0000 7f5a1a092640 10 monclient(hunting): authenticate will time out at 1053.540304s 2024-03-29T10:39:11.263 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.280+0000 7f5a19090640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f5a140a9750 0x7f5a14143710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:60848/0 (socket says 172.21.15.84:60848) 2024-03-29T10:39:11.264 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.280+0000 7f5a19090640 1 -- 172.21.15.84:0/2178328268 learned_addr learned my addr 172.21.15.84:0/2178328268 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:39:11.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.282+0000 7f5a13fff640 1 -- 172.21.15.84:0/2178328268 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493109825 0 0) 0x7f5a140af880 con 0x7f5a140a9750 2024-03-29T10:39:11.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.282+0000 7f5a13fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:11.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.282+0000 7f5a13fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:11.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.282+0000 7f5a13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:11.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.282+0000 7f5a13fff640 10 monclient(hunting): my global_id is 4412 2024-03-29T10:39:11.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.282+0000 7f5a13fff640 1 -- 172.21.15.84:0/2178328268 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59fc003520 con 0x7f5a140a9750 2024-03-29T10:39:11.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.283+0000 7f5a13fff640 1 -- 172.21.15.84:0/2178328268 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 128953468 0 0) 0x7f5a140b0070 con 0x7f5a140adca0 2024-03-29T10:39:11.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.283+0000 7f5a13fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:11.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.283+0000 7f5a13fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:11.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.283+0000 7f5a13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:11.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.283+0000 7f5a13fff640 10 monclient(hunting): my global_id is 4366 2024-03-29T10:39:11.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.283+0000 7f5a13fff640 1 -- 172.21.15.84:0/2178328268 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a140af880 con 0x7f5a140adca0 2024-03-29T10:39:11.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.284+0000 7f5a13fff640 1 -- 172.21.15.84:0/2178328268 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 251749157 0 0) 0x7f59fc003520 con 0x7f5a140a9750 2024-03-29T10:39:11.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.284+0000 7f5a13fff640 1 -- 172.21.15.84:0/2178328268 >> v1:172.21.15.86:6789/0 conn(0x7f5a140adca0 legacy=0x7f5a14146370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:11.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.284+0000 7f5a13fff640 1 -- 172.21.15.84:0/2178328268 >> v1:172.21.15.84:6789/0 conn(0x7f5a14146a80 legacy=0x7f5a14151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:11.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.284+0000 7f5a13fff640 10 monclient: _finish_hunting 0 2024-03-29T10:39:11.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.284+0000 7f5a13fff640 1 monclient: found mon.noname-b 2024-03-29T10:39:11.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.284+0000 7f5a13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:11.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.284+0000 7f5a13fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:39:11.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.284+0000 7f5a13fff640 1 -- 172.21.15.84:0/2178328268 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a140ac030 con 0x7f5a140a9750 2024-03-29T10:39:11.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.284+0000 7f5a13fff640 10 monclient: _finish_auth 0 2024-03-29T10:39:11.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.284+0000 7f5a13fff640 10 monclient: _check_auth_tickets 2024-03-29T10:39:11.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.284+0000 7f5a13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:11.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.284+0000 7f5a13fff640 1 -- 172.21.15.84:0/2178328268 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f5a08001690 con 0x7f5a140a9750 2024-03-29T10:39:11.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.284+0000 7f5a13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:11.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.284+0000 7f5a13fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:11.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.284+0000 7f5a13fff640 10 monclient: dump: 2024-03-29T10:39:11.273 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:11.273 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:11.273 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:11.273 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:11.273 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:11.273 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:11.273 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:11.273 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:11.273 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:11.274 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:11.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.284+0000 7f5a1a092640 5 monclient: authenticate success, global_id 4412 2024-03-29T10:39:11.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.284+0000 7f5a1a092640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:11.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.286+0000 7f5a13fff640 1 -- 172.21.15.84:0/2178328268 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a080017f0 con 0x7f5a140a9750 2024-03-29T10:39:11.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.286+0000 7f5a13fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:11.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.286+0000 7f5a13fff640 1 -- 172.21.15.84:0/2178328268 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f5a08001c00 con 0x7f5a140a9750 2024-03-29T10:39:11.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.286+0000 7f5a13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:11.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.286+0000 7f5a13fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:11.274 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.286+0000 7f5a13fff640 10 monclient: dump: 2024-03-29T10:39:11.274 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:11.274 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:11.274 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:11.274 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:11.274 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:11.275 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:11.275 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:11.275 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:11.275 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:11.275 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:11.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.286+0000 7f5a1a092640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:11.275 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.286+0000 7f5a1a092640 4 set_mon_vals no callback set 2024-03-29T10:39:11.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.287+0000 7f5a1a092640 10 monclient: shutdown 2024-03-29T10:39:11.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.287+0000 7f5a1a092640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:11.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.287+0000 7f5a1a092640 1 -- 172.21.15.84:0/2178328268 >> v1:172.21.15.84:6790/0 conn(0x7f5a140a9750 legacy=0x7f5a14143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:11.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.288+0000 7f5a1a092640 1 -- 172.21.15.84:0/2178328268 shutdown_connections 2024-03-29T10:39:11.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.288+0000 7f5a1a092640 1 -- 172.21.15.84:0/2178328268 >> 172.21.15.84:0/2178328268 conn(0x7f5a140673e0 msgr2=0x7f5a140b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:11.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.288+0000 7f5a1a092640 1 -- 172.21.15.84:0/2178328268 shutdown_connections 2024-03-29T10:39:11.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.288+0000 7f5a1a092640 1 -- 172.21.15.84:0/2178328268 wait complete. 2024-03-29T10:39:11.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.289+0000 7f5a1a092640 10 monclient: build_initial_monmap 2024-03-29T10:39:11.277 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.289+0000 7f5a1a092640 10 monclient: monmap: 2024-03-29T10:39:11.277 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:39:11.277 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:11.277 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:39:11.277 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:39:11.277 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:11.277 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:11.277 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:39:11.277 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:39:11.278 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:39:11.278 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:11.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.290+0000 7f5a1a092640 1 librados: starting msgr at 2024-03-29T10:39:11.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.290+0000 7f5a1a092640 1 librados: starting objecter 2024-03-29T10:39:11.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.290+0000 7f5a1a092640 1 Processor -- start 2024-03-29T10:39:11.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.292+0000 7f5a1a092640 1 -- start start 2024-03-29T10:39:11.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.292+0000 7f5a1a092640 1 librados: setting wanted keys 2024-03-29T10:39:11.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.292+0000 7f5a1a092640 1 librados: calling monclient init 2024-03-29T10:39:11.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.292+0000 7f5a1a092640 10 monclient: init 2024-03-29T10:39:11.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.293+0000 7f5a1a092640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:11.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.293+0000 7f5a1a092640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:39:11.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.293+0000 7f5a1a092640 10 monclient(hunting): picked mon.noname-b con 0x7f5a140a9750 addr v1:172.21.15.86:6789/0 2024-03-29T10:39:11.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.293+0000 7f5a1a092640 10 monclient(hunting): picked mon.noname-a con 0x7f5a140adca0 addr v1:172.21.15.84:6789/0 2024-03-29T10:39:11.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.293+0000 7f5a1a092640 10 monclient(hunting): picked mon.noname-c con 0x7f5a14146a80 addr v1:172.21.15.84:6790/0 2024-03-29T10:39:11.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.293+0000 7f5a1a092640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a141db560 con 0x7f5a140adca0 2024-03-29T10:39:11.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.293+0000 7f5a1a092640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a14070e40 con 0x7f5a140a9750 2024-03-29T10:39:11.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.293+0000 7f5a1a092640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a14071fc0 con 0x7f5a14146a80 2024-03-29T10:39:11.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.293+0000 7f5a1a092640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:11.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.293+0000 7f5a1a092640 10 monclient(hunting): authenticate will time out at 1053.556738s 2024-03-29T10:39:11.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.297+0000 7f5a19090640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f5a140a9750 0x7f5a140718b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:36403/0 (socket says 172.21.15.84:36403) 2024-03-29T10:39:11.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.297+0000 7f5a19090640 1 -- 172.21.15.84:0/3171898092 learned_addr learned my addr 172.21.15.84:0/3171898092 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:39:11.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.297+0000 7f5a11ffb640 10 client.?.objecter ms_handle_connect 0x7f5a140a9750 2024-03-29T10:39:11.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.297+0000 7f5a11ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:11.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.297+0000 7f5a11ffb640 1 -- 172.21.15.84:0/3171898092 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2605959468 0 0) 0x7f5a14070e40 con 0x7f5a140a9750 2024-03-29T10:39:11.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.297+0000 7f5a11ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:11.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.297+0000 7f5a11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:11.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.297+0000 7f5a11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:11.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.297+0000 7f5a11ffb640 10 monclient(hunting): my global_id is 4369 2024-03-29T10:39:11.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.297+0000 7f5a11ffb640 1 -- 172.21.15.84:0/3171898092 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a00004eb0 con 0x7f5a140a9750 2024-03-29T10:39:11.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.298+0000 7f5a11ffb640 1 -- 172.21.15.84:0/3171898092 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2454948790 0 0) 0x7f5a00004eb0 con 0x7f5a140a9750 2024-03-29T10:39:11.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.298+0000 7f5a11ffb640 1 -- 172.21.15.84:0/3171898092 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a14070e40 con 0x7f5a140a9750 2024-03-29T10:39:11.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.298+0000 7f5a11ffb640 1 -- 172.21.15.84:0/3171898092 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f5a08001c40 con 0x7f5a140a9750 2024-03-29T10:39:11.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.298+0000 7f5a11ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:11.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.298+0000 7f5a11ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:11.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.298+0000 7f5a11ffb640 10 monclient(hunting): dump: 2024-03-29T10:39:11.285 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:11.285 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:11.285 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:11.285 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:11.285 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:11.285 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:11.285 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:11.285 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:11.286 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:11.286 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:11.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.299+0000 7f5a11ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:11.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.299+0000 7f5a11ffb640 1 -- 172.21.15.84:0/3171898092 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 989679753 0 0) 0x7f5a14070e40 con 0x7f5a140a9750 2024-03-29T10:39:11.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.299+0000 7f5a11ffb640 1 -- 172.21.15.84:0/3171898092 >> v1:172.21.15.84:6790/0 conn(0x7f5a14146a80 legacy=0x7f5a1413e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:11.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.299+0000 7f5a11ffb640 1 -- 172.21.15.84:0/3171898092 >> v1:172.21.15.84:6789/0 conn(0x7f5a140adca0 legacy=0x7f5a141dba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:11.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.299+0000 7f5a11ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:39:11.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.299+0000 7f5a11ffb640 1 monclient: found mon.b 2024-03-29T10:39:11.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.299+0000 7f5a11ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:11.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.299+0000 7f5a11ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:39:11.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.299+0000 7f5a11ffb640 1 -- 172.21.15.84:0/3171898092 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a140722a0 con 0x7f5a140a9750 2024-03-29T10:39:11.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.299+0000 7f5a11ffb640 10 monclient: _finish_auth 0 2024-03-29T10:39:11.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.299+0000 7f5a11ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:39:11.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.299+0000 7f5a11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:11.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.299+0000 7f5a1a092640 5 monclient: authenticate success, global_id 4369 2024-03-29T10:39:11.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.299+0000 7f5a1a092640 10 monclient: _renew_subs 2024-03-29T10:39:11.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.299+0000 7f5a1a092640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:39:11.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.299+0000 7f5a1a092640 1 -- 172.21.15.84:0/3171898092 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a1406e4d0 con 0x7f5a140a9750 2024-03-29T10:39:11.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.300+0000 7f5a11ffb640 1 -- 172.21.15.84:0/3171898092 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a08004a90 con 0x7f5a140a9750 2024-03-29T10:39:11.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.300+0000 7f5a11ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:11.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.300+0000 7f5a137fe640 4 set_mon_vals no callback set 2024-03-29T10:39:11.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.301+0000 7f5a11ffb640 1 -- 172.21.15.84:0/3171898092 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f5a08004ea0 con 0x7f5a140a9750 2024-03-29T10:39:11.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.301+0000 7f5a11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:11.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.301+0000 7f5a11ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:11.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.301+0000 7f5a11ffb640 10 monclient: dump: 2024-03-29T10:39:11.289 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:11.289 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:11.289 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:11.289 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:11.289 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:11.289 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:11.289 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:11.289 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:11.290 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:11.290 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:11.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.301+0000 7f5a19090640 1 -- 172.21.15.84:0/3171898092 >> v1:172.21.15.86:6789/0 conn(0x7f5a140a9750 legacy=0x7f5a140718b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-29T10:39:11.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.301+0000 7f5a19090640 1 -- 172.21.15.84:0/3171898092 >> v1:172.21.15.86:6789/0 conn(0x7f5a140a9750 legacy=0x7f5a140718b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:39:11.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.301+0000 7f5a19090640 1 --1- 172.21.15.84:0/3171898092 >> v1:172.21.15.86:6789/0 conn(0x7f5a140a9750 0x7f5a140718b0 :-1 s=OPENED pgs=516931 cs=1 l=1).handle_message read tag failed 2024-03-29T10:39:11.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.301+0000 7f5a19090640 1 --1- 172.21.15.84:0/3171898092 >> v1:172.21.15.86:6789/0 conn(0x7f5a140a9750 0x7f5a140718b0 :-1 s=OPENED pgs=516931 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:39:11.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.301+0000 7f5a11ffb640 10 monclient: ms_handle_reset current mon v1:172.21.15.86:6789/0 2024-03-29T10:39:11.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.301+0000 7f5a11ffb640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:11.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.301+0000 7f5a11ffb640 1 -- 172.21.15.84:0/3171898092 >> v1:172.21.15.86:6789/0 conn(0x7f5a140a9750 legacy=0x7f5a140718b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:39:11.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.301+0000 7f5a11ffb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:39:11.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.301+0000 7f5a11ffb640 10 monclient(hunting): picked mon.b con 0x7f5a00006150 addr v1:172.21.15.86:6789/0 2024-03-29T10:39:11.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.301+0000 7f5a11ffb640 10 monclient(hunting): picked mon.a con 0x7f5a140adca0 addr v1:172.21.15.84:6789/0 2024-03-29T10:39:11.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.302+0000 7f5a11ffb640 10 monclient(hunting): picked mon.c con 0x7f5a14146a80 addr v1:172.21.15.84:6790/0 2024-03-29T10:39:11.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.302+0000 7f5a11ffb640 1 -- 172.21.15.84:0/3171898092 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5a14070e40 con 0x7f5a140adca0 2024-03-29T10:39:11.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.302+0000 7f5a11ffb640 1 -- 172.21.15.84:0/3171898092 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5a141db560 con 0x7f5a00006150 2024-03-29T10:39:11.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.302+0000 7f5a11ffb640 1 -- 172.21.15.84:0/3171898092 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f5a14071fc0 con 0x7f5a14146a80 2024-03-29T10:39:11.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.302+0000 7f5a11ffb640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:11.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.301+0000 7f5a1a092640 10 client.4369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:11.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.302+0000 7f5a1a092640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:11.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.302+0000 7f5a1a092640 1 librados: init done 2024-03-29T10:39:11.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.302+0000 7f5a11ffb640 10 client.4369.objecter ms_handle_connect 0x7f5a14146a80 2024-03-29T10:39:11.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.302+0000 7f5a11ffb640 10 client.4369.objecter resend_mon_ops 2024-03-29T10:39:11.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.302+0000 7f5a11ffb640 1 -- 172.21.15.84:0/3171898092 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825624512 0 0) 0x7f5a14071fc0 con 0x7f5a14146a80 2024-03-29T10:39:11.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.302+0000 7f5a11ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:11.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.302+0000 7f5a11ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:11.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.302+0000 7f5a11ffb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T10:39:11.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.302+0000 7f5a11ffb640 1 -- 172.21.15.84:0/3171898092 --> v1:172.21.15.84:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f5a00013970 con 0x7f5a14146a80 2024-03-29T10:39:11.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.304+0000 7f5a11ffb640 1 -- 172.21.15.84:0/3171898092 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2909419214 0 0) 0x7f5a00013970 con 0x7f5a14146a80 2024-03-29T10:39:11.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.304+0000 7f5a11ffb640 1 -- 172.21.15.84:0/3171898092 >> v1:172.21.15.86:6789/0 conn(0x7f5a00006150 legacy=0x7f5a00006530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:39:11.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.304+0000 7f5a1a092640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:11.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.305+0000 7f5a11ffb640 1 -- 172.21.15.84:0/3171898092 >> v1:172.21.15.84:6789/0 conn(0x7f5a140adca0 legacy=0x7f5a0000c310 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:39:11.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.306+0000 7f5a11ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:39:11.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.306+0000 7f5a11ffb640 1 monclient: found mon.c 2024-03-29T10:39:11.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.306+0000 7f5a11ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:11.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.306+0000 7f5a11ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:39:11.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.306+0000 7f5a11ffb640 1 -- 172.21.15.84:0/3171898092 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f5a00004eb0 con 0x7f5a14146a80 2024-03-29T10:39:11.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.306+0000 7f5a11ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:39:11.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.306+0000 7f5a11ffb640 1 -- 172.21.15.84:0/3171898092 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a141dc370 con 0x7f5a14146a80 2024-03-29T10:39:11.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.306+0000 7f5a11ffb640 10 monclient: _finish_auth 0 2024-03-29T10:39:11.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.306+0000 7f5a11ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:39:11.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.306+0000 7f5a11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:11.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.306+0000 7f5a11ffb640 1 -- 172.21.15.84:0/3171898092 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f5a04001850 con 0x7f5a14146a80 2024-03-29T10:39:11.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.306+0000 7f5a11ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:11.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.306+0000 7f5a11ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:11.294 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.306+0000 7f5a11ffb640 10 monclient: dump: 2024-03-29T10:39:11.294 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:11.294 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:11.294 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:11.294 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:11.294 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:11.294 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:11.294 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:11.294 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:11.294 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:11.294 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:11.294 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.307+0000 7f5a1a092640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:11.294 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.307+0000 7f5a1a092640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:39:11.294 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.307+0000 7f5a1a092640 1 -- 172.21.15.84:0/3171898092 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a1404f730 con 0x7f5a14146a80 2024-03-29T10:39:11.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.311+0000 7f5a11ffb640 1 -- 172.21.15.84:0/3171898092 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a040019b0 con 0x7f5a14146a80 2024-03-29T10:39:11.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.311+0000 7f5a11ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:11.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.311+0000 7f5a11ffb640 1 -- 172.21.15.84:0/3171898092 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f5a04001c60 con 0x7f5a14146a80 2024-03-29T10:39:11.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.312+0000 7f5a12ffd640 4 set_mon_vals no callback set 2024-03-29T10:39:11.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.313+0000 7f5a11ffb640 1 -- 172.21.15.84:0/3171898092 <== mon.2 v1:172.21.15.84:6790/0 6 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f5a04083130 con 0x7f5a14146a80 2024-03-29T10:39:11.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.313+0000 7f5a11ffb640 10 client.4369.objecter ms_dispatch 0x7f5a14000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:11.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.313+0000 7f5a11ffb640 3 client.4369.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:11.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.313+0000 7f5a11ffb640 3 client.4369.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:11.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.313+0000 7f5a11ffb640 20 client.4369.objecter dump_active .. 0 homeless 2024-03-29T10:39:11.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.315+0000 7f5a19090640 10 monclient: get_auth_request con 0x7f5a140a9750 auth_method 0 2024-03-29T10:39:11.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.315+0000 7f5a11ffb640 10 client.4369.objecter ms_handle_connect 0x7f5a140a9750 2024-03-29T10:39:11.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.320+0000 7f5a11ffb640 1 -- 172.21.15.84:0/3171898092 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5a0404ea90 con 0x7f5a14146a80 2024-03-29T10:39:11.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.320+0000 7f5a11ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:11.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.320+0000 7f5a11ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:11.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.682+0000 7f5a1a092640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:11.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.682+0000 7f5a1a092640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:11.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.682+0000 7f5a1a092640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:39:11.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.682+0000 7f5a1a092640 1 -- 172.21.15.84:0/3171898092 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a14074e10 con 0x7f5a14146a80 2024-03-29T10:39:11.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.682+0000 7f5a11ffb640 1 -- 172.21.15.84:0/3171898092 <== mon.2 v1:172.21.15.84:6790/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f5a040509c0 con 0x7f5a14146a80 2024-03-29T10:39:11.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.682+0000 7f5a11ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:11.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.682+0000 7f5a11ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:11.668 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:39:11.668 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:11.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.687+0000 7f5a1a092640 10 librados: watch_flush enter 2024-03-29T10:39:11.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.687+0000 7f5a1a092640 10 librados: watch_flush exit 2024-03-29T10:39:11.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.687+0000 7f5a1a092640 20 client.4369.objecter shutdown clearing up homeless session... 2024-03-29T10:39:11.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.687+0000 7f5a1a092640 10 client.4369.objecter successfully canceled tick 2024-03-29T10:39:11.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.687+0000 7f5a1a092640 1 -- 172.21.15.84:0/3171898092 >> v1:172.21.15.86:6816/3337225337 conn(0x7f5a140a9750 legacy=0x7f5a0007b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:11.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.687+0000 7f5a1a092640 10 monclient: shutdown 2024-03-29T10:39:11.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.687+0000 7f5a1a092640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:11.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.687+0000 7f5a1a092640 1 -- 172.21.15.84:0/3171898092 >> v1:172.21.15.84:6790/0 conn(0x7f5a14146a80 legacy=0x7f5a0000fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:11.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.688+0000 7f5a1a092640 1 -- 172.21.15.84:0/3171898092 shutdown_connections 2024-03-29T10:39:11.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.688+0000 7f5a1a092640 1 -- 172.21.15.84:0/3171898092 >> 172.21.15.84:0/3171898092 conn(0x7f5a140673e0 msgr2=0x7f5a140aa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:11.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.690+0000 7f5a1a092640 1 -- 172.21.15.84:0/3171898092 shutdown_connections 2024-03-29T10:39:11.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.691+0000 7f5a1a092640 1 -- 172.21.15.84:0/3171898092 wait complete. 2024-03-29T10:39:11.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:11.692+0000 7f5a1a092640 1 librados: shutdown 2024-03-29T10:39:11.693 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:18.500 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:18.661 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.677+0000 7f2076a99640 10 monclient: get_monmap_and_config 2024-03-29T10:39:18.662 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.677+0000 7f2076a99640 10 monclient: build_initial_monmap 2024-03-29T10:39:18.662 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.677+0000 7f2076a99640 10 monclient: monmap: 2024-03-29T10:39:18.662 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:39:18.662 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:18.662 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:39:18.678845+0000 2024-03-29T10:39:18.662 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:39:18.678845+0000 2024-03-29T10:39:18.662 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:18.662 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:18.662 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:39:18.662 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:39:18.662 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:39:18.663 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:18.663 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.679+0000 7f2076a99640 1 Processor -- start 2024-03-29T10:39:18.663 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.679+0000 7f2076a99640 1 -- start start 2024-03-29T10:39:18.663 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.679+0000 7f2076a99640 10 monclient: init 2024-03-29T10:39:18.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.680+0000 7f2076a99640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:18.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.680+0000 7f2076a99640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:39:18.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.680+0000 7f2076a99640 10 monclient(hunting): picked mon.noname-b con 0x7f2068036a60 addr v1:172.21.15.84:6790/0 2024-03-29T10:39:18.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.681+0000 7f2076a99640 10 monclient(hunting): picked mon.noname-c con 0x7f20680c5de0 addr v1:172.21.15.86:6789/0 2024-03-29T10:39:18.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.681+0000 7f2076a99640 10 monclient(hunting): picked mon.noname-a con 0x7f20680c3070 addr v1:172.21.15.84:6789/0 2024-03-29T10:39:18.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.681+0000 7f2076a99640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f206802cc60 con 0x7f20680c3070 2024-03-29T10:39:18.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.681+0000 7f2076a99640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f206802e040 con 0x7f20680c5de0 2024-03-29T10:39:18.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.681+0000 7f2076a99640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f206802e320 con 0x7f2068036a60 2024-03-29T10:39:18.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.681+0000 7f2076a99640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:18.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.681+0000 7f2076a99640 10 monclient(hunting): authenticate will time out at 1060.944572s 2024-03-29T10:39:18.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.681+0000 7f2075a97640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f2068036a60 0x7f20680c56d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:40018/0 (socket says 172.21.15.84:40018) 2024-03-29T10:39:18.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.681+0000 7f2075a97640 1 -- 172.21.15.84:0/3406896919 learned_addr learned my addr 172.21.15.84:0/3406896919 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:39:18.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.685+0000 7f2074a95640 1 -- 172.21.15.84:0/3406896919 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 201779815 0 0) 0x7f206802cc60 con 0x7f20680c3070 2024-03-29T10:39:18.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.685+0000 7f2074a95640 10 monclient(hunting): _negotiate 2024-03-29T10:39:18.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.685+0000 7f2074a95640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:18.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.685+0000 7f2074a95640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:18.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.685+0000 7f2074a95640 10 monclient(hunting): my global_id is 4398 2024-03-29T10:39:18.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.685+0000 7f2074a95640 1 -- 172.21.15.84:0/3406896919 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f205c003520 con 0x7f20680c3070 2024-03-29T10:39:18.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.685+0000 7f2074a95640 1 -- 172.21.15.84:0/3406896919 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532065465 0 0) 0x7f206802e320 con 0x7f2068036a60 2024-03-29T10:39:18.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.685+0000 7f2074a95640 10 monclient(hunting): _negotiate 2024-03-29T10:39:18.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.685+0000 7f2074a95640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:18.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.685+0000 7f2074a95640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:18.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.685+0000 7f2074a95640 10 monclient(hunting): my global_id is 4415 2024-03-29T10:39:18.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.685+0000 7f2074a95640 1 -- 172.21.15.84:0/3406896919 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f206802cc60 con 0x7f2068036a60 2024-03-29T10:39:18.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.685+0000 7f2074a95640 1 -- 172.21.15.84:0/3406896919 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 282082410 0 0) 0x7f205c003520 con 0x7f20680c3070 2024-03-29T10:39:18.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.685+0000 7f2074a95640 1 -- 172.21.15.84:0/3406896919 >> v1:172.21.15.84:6790/0 conn(0x7f2068036a60 legacy=0x7f20680c56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:18.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.685+0000 7f2074a95640 1 -- 172.21.15.84:0/3406896919 >> v1:172.21.15.86:6789/0 conn(0x7f20680c5de0 legacy=0x7f20680c2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:18.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.685+0000 7f2074a95640 10 monclient: _finish_hunting 0 2024-03-29T10:39:18.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.685+0000 7f2074a95640 1 monclient: found mon.noname-a 2024-03-29T10:39:18.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.685+0000 7f2074a95640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:18.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.685+0000 7f2074a95640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T10:39:18.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.685+0000 7f2074a95640 1 -- 172.21.15.84:0/3406896919 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f206802c830 con 0x7f20680c3070 2024-03-29T10:39:18.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.686+0000 7f2074a95640 10 monclient: _finish_auth 0 2024-03-29T10:39:18.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.686+0000 7f2074a95640 10 monclient: _check_auth_tickets 2024-03-29T10:39:18.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.686+0000 7f2074a95640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:18.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.686+0000 7f2074a95640 1 -- 172.21.15.84:0/3406896919 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f207004f730 con 0x7f20680c3070 2024-03-29T10:39:18.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.686+0000 7f2074a95640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:18.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.686+0000 7f2074a95640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:39:18.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.686+0000 7f2074a95640 10 monclient: dump: 2024-03-29T10:39:18.674 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:18.674 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:18.674 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:18.674 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:18.674 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:18.674 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:18.674 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:18.674 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:18.675 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:18.675 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:18.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.686+0000 7f2076a99640 5 monclient: authenticate success, global_id 4398 2024-03-29T10:39:18.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.686+0000 7f2076a99640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:18.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.686+0000 7f2074a95640 1 -- 172.21.15.84:0/3406896919 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2070075f10 con 0x7f20680c3070 2024-03-29T10:39:18.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.686+0000 7f2074a95640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:18.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.686+0000 7f2074a95640 1 -- 172.21.15.84:0/3406896919 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f20700758b0 con 0x7f20680c3070 2024-03-29T10:39:18.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.686+0000 7f2076a99640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:18.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.686+0000 7f2076a99640 4 set_mon_vals no callback set 2024-03-29T10:39:18.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.688+0000 7f2076a99640 10 monclient: shutdown 2024-03-29T10:39:18.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.688+0000 7f2074a95640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:18.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.688+0000 7f2074a95640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:18.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.688+0000 7f2074a95640 10 monclient: dump: 2024-03-29T10:39:18.675 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:18.675 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:18.676 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:18.676 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:18.676 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:18.676 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:18.676 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:18.676 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:18.676 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:18.676 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:18.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.688+0000 7f2076a99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:18.676 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.688+0000 7f2076a99640 1 -- 172.21.15.84:0/3406896919 >> v1:172.21.15.84:6789/0 conn(0x7f20680c3070 legacy=0x7f20680d0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:18.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.689+0000 7f2076a99640 1 -- 172.21.15.84:0/3406896919 shutdown_connections 2024-03-29T10:39:18.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.689+0000 7f2076a99640 1 -- 172.21.15.84:0/3406896919 >> 172.21.15.84:0/3406896919 conn(0x7f2068005c00 msgr2=0x7f2068006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:18.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.690+0000 7f2076a99640 1 -- 172.21.15.84:0/3406896919 shutdown_connections 2024-03-29T10:39:18.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.690+0000 7f2076a99640 1 -- 172.21.15.84:0/3406896919 wait complete. 2024-03-29T10:39:18.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.691+0000 7f2076a99640 10 monclient: build_initial_monmap 2024-03-29T10:39:18.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.691+0000 7f2076a99640 10 monclient: monmap: 2024-03-29T10:39:18.679 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:39:18.679 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:18.679 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:39:18.679 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:39:18.679 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:18.679 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:18.679 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:39:18.680 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:39:18.680 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:39:18.680 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:18.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.692+0000 7f2076a99640 1 librados: starting msgr at 2024-03-29T10:39:18.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.692+0000 7f2076a99640 1 librados: starting objecter 2024-03-29T10:39:18.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.692+0000 7f2076a99640 1 Processor -- start 2024-03-29T10:39:18.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.692+0000 7f2076a99640 1 -- start start 2024-03-29T10:39:18.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.692+0000 7f2076a99640 1 librados: setting wanted keys 2024-03-29T10:39:18.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.692+0000 7f2076a99640 1 librados: calling monclient init 2024-03-29T10:39:18.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.692+0000 7f2076a99640 10 monclient: init 2024-03-29T10:39:18.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.693+0000 7f2076a99640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:18.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.693+0000 7f2076a99640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:39:18.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.693+0000 7f2076a99640 10 monclient(hunting): picked mon.noname-b con 0x7f2068036a60 addr v1:172.21.15.86:6789/0 2024-03-29T10:39:18.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.693+0000 7f2076a99640 10 monclient(hunting): picked mon.noname-a con 0x7f20680c3070 addr v1:172.21.15.84:6789/0 2024-03-29T10:39:18.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.693+0000 7f2076a99640 10 monclient(hunting): picked mon.noname-c con 0x7f206803a270 addr v1:172.21.15.84:6790/0 2024-03-29T10:39:18.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.693+0000 7f2076a99640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20681656a0 con 0x7f20680c3070 2024-03-29T10:39:18.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.693+0000 7f2076a99640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20680092b0 con 0x7f2068036a60 2024-03-29T10:39:18.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.693+0000 7f2076a99640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20680c52a0 con 0x7f206803a270 2024-03-29T10:39:18.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.693+0000 7f2076a99640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:18.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.693+0000 7f2076a99640 10 monclient(hunting): authenticate will time out at 1060.957074s 2024-03-29T10:39:18.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.694+0000 7f2076298640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f206803a270 0x7f2068164f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:40240/0 (socket says 172.21.15.84:40240) 2024-03-29T10:39:18.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.694+0000 7f2076298640 1 -- 172.21.15.84:0/260744131 learned_addr learned my addr 172.21.15.84:0/260744131 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:39:18.681 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.694+0000 7f20667fc640 10 client.?.objecter ms_handle_connect 0x7f206803a270 2024-03-29T10:39:18.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.694+0000 7f20667fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:18.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.694+0000 7f20667fc640 1 -- 172.21.15.84:0/260744131 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3150205411 0 0) 0x7f20680c52a0 con 0x7f206803a270 2024-03-29T10:39:18.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.695+0000 7f20667fc640 10 monclient(hunting): _negotiate 2024-03-29T10:39:18.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.695+0000 7f20667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:18.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.695+0000 7f20667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:18.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.695+0000 7f20667fc640 10 monclient(hunting): my global_id is 4418 2024-03-29T10:39:18.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.695+0000 7f20667fc640 1 -- 172.21.15.84:0/260744131 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2054004eb0 con 0x7f206803a270 2024-03-29T10:39:18.683 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.695+0000 7f20667fc640 10 client.?.objecter ms_handle_connect 0x7f20680c3070 2024-03-29T10:39:18.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.695+0000 7f20667fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:18.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.695+0000 7f20667fc640 1 -- 172.21.15.84:0/260744131 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2621348745 0 0) 0x7f2054004eb0 con 0x7f206803a270 2024-03-29T10:39:18.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.695+0000 7f20667fc640 1 -- 172.21.15.84:0/260744131 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20680c52a0 con 0x7f206803a270 2024-03-29T10:39:18.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.696+0000 7f20667fc640 1 -- 172.21.15.84:0/260744131 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f207004f400 con 0x7f206803a270 2024-03-29T10:39:18.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.696+0000 7f20667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:18.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.696+0000 7f20667fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:18.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.696+0000 7f20667fc640 10 monclient(hunting): dump: 2024-03-29T10:39:18.684 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:18.684 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:18.684 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:18.684 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:18.684 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:18.684 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:18.685 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:18.685 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:18.685 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:18.685 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:18.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.696+0000 7f20667fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:18.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.696+0000 7f20667fc640 1 -- 172.21.15.84:0/260744131 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2311835601 0 0) 0x7f20680c52a0 con 0x7f206803a270 2024-03-29T10:39:18.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.696+0000 7f20667fc640 1 -- 172.21.15.84:0/260744131 >> v1:172.21.15.86:6789/0 conn(0x7f2068036a60 legacy=0x7f20680c38b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:39:18.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.698+0000 7f20667fc640 1 -- 172.21.15.84:0/260744131 >> v1:172.21.15.84:6789/0 conn(0x7f20680c3070 legacy=0x7f2068164880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:18.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.698+0000 7f20667fc640 10 monclient: _finish_hunting 0 2024-03-29T10:39:18.685 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.698+0000 7f20667fc640 1 monclient: found mon.c 2024-03-29T10:39:18.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.698+0000 7f20667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:18.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.698+0000 7f20667fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:39:18.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.698+0000 7f20667fc640 1 -- 172.21.15.84:0/260744131 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20680c5580 con 0x7f206803a270 2024-03-29T10:39:18.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.698+0000 7f20667fc640 10 monclient: _finish_auth 0 2024-03-29T10:39:18.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.698+0000 7f20667fc640 10 monclient: _check_auth_tickets 2024-03-29T10:39:18.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.698+0000 7f20667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:18.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.698+0000 7f2076a99640 5 monclient: authenticate success, global_id 4418 2024-03-29T10:39:18.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.698+0000 7f2076a99640 10 monclient: _renew_subs 2024-03-29T10:39:18.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.698+0000 7f2076a99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:39:18.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.698+0000 7f2076a99640 1 -- 172.21.15.84:0/260744131 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20680c5860 con 0x7f206803a270 2024-03-29T10:39:18.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.698+0000 7f2076a99640 10 client.4418.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:18.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.698+0000 7f2076a99640 10 monclient: _renew_subs 2024-03-29T10:39:18.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.698+0000 7f2076a99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:39:18.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.698+0000 7f2076a99640 1 -- 172.21.15.84:0/260744131 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f20680c4830 con 0x7f206803a270 2024-03-29T10:39:18.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.698+0000 7f2076a99640 1 librados: init done 2024-03-29T10:39:18.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.698+0000 7f20667fc640 1 -- 172.21.15.84:0/260744131 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f207004f730 con 0x7f206803a270 2024-03-29T10:39:18.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.698+0000 7f20667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:18.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.698+0000 7f20667fc640 1 -- 172.21.15.84:0/260744131 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f2070067bf0 con 0x7f206803a270 2024-03-29T10:39:18.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.698+0000 7f20667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:18.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.698+0000 7f20667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:18.692 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.698+0000 7f20667fc640 10 monclient: dump: 2024-03-29T10:39:18.692 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:18.693 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:18.693 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:18.693 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:18.693 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:18.693 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:18.693 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:18.693 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:18.693 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:18.693 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:18.693 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.699+0000 7f2076a99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:18.693 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.699+0000 7f2076a99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:18.693 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.699+0000 7f2076a99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:39:18.693 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.699+0000 7f2076a99640 1 -- 172.21.15.84:0/260744131 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2068005120 con 0x7f206803a270 2024-03-29T10:39:18.693 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.700+0000 7f20667fc640 1 -- 172.21.15.84:0/260744131 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f2070067d50 con 0x7f206803a270 2024-03-29T10:39:18.694 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.701+0000 7f20667fc640 1 -- 172.21.15.84:0/260744131 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f20700ff650 con 0x7f206803a270 2024-03-29T10:39:18.694 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.701+0000 7f20667fc640 10 client.4418.objecter ms_dispatch 0x7f2070000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:18.694 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.701+0000 7f20667fc640 3 client.4418.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:18.694 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.701+0000 7f20667fc640 3 client.4418.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:18.694 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.701+0000 7f20667fc640 20 client.4418.objecter dump_active .. 0 homeless 2024-03-29T10:39:18.694 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.701+0000 7f2067fff640 4 set_mon_vals no callback set 2024-03-29T10:39:18.694 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.702+0000 7f2075a97640 10 monclient: get_auth_request con 0x7f205406da60 auth_method 0 2024-03-29T10:39:18.694 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.703+0000 7f20667fc640 10 client.4418.objecter ms_handle_connect 0x7f205406da60 2024-03-29T10:39:18.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.713+0000 7f20667fc640 1 -- 172.21.15.84:0/260744131 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f20700caf30 con 0x7f206803a270 2024-03-29T10:39:18.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.713+0000 7f20667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:18.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:18.713+0000 7f20667fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:19.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:19.112+0000 7f2076a99640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:19.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:19.112+0000 7f2076a99640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:19.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:19.112+0000 7f2076a99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:39:19.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:19.112+0000 7f2076a99640 1 -- 172.21.15.84:0/260744131 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20680c35d0 con 0x7f206803a270 2024-03-29T10:39:19.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:19.113+0000 7f20667fc640 1 -- 172.21.15.84:0/260744131 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f20680c35d0 con 0x7f206803a270 2024-03-29T10:39:19.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:19.113+0000 7f20667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:19.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:19.113+0000 7f20667fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:19.097 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:39:19.097 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:19.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:19.117+0000 7f2076a99640 10 librados: watch_flush enter 2024-03-29T10:39:19.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:19.117+0000 7f2076a99640 10 librados: watch_flush exit 2024-03-29T10:39:19.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:19.117+0000 7f2076a99640 20 client.4418.objecter shutdown clearing up homeless session... 2024-03-29T10:39:19.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:19.117+0000 7f2076a99640 10 client.4418.objecter successfully canceled tick 2024-03-29T10:39:19.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:19.117+0000 7f2076a99640 1 -- 172.21.15.84:0/260744131 >> v1:172.21.15.86:6816/3337225337 conn(0x7f205406da60 legacy=0x7f205406ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:19.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:19.117+0000 7f2076a99640 10 monclient: shutdown 2024-03-29T10:39:19.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:19.117+0000 7f2076a99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:19.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:19.117+0000 7f2076a99640 1 -- 172.21.15.84:0/260744131 >> v1:172.21.15.84:6790/0 conn(0x7f206803a270 legacy=0x7f2068164f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:19.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:19.117+0000 7f2076a99640 1 -- 172.21.15.84:0/260744131 shutdown_connections 2024-03-29T10:39:19.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:19.117+0000 7f2076a99640 1 -- 172.21.15.84:0/260744131 >> 172.21.15.84:0/260744131 conn(0x7f2068005c00 msgr2=0x7f20680bfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:19.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:19.119+0000 7f2076a99640 1 -- 172.21.15.84:0/260744131 shutdown_connections 2024-03-29T10:39:19.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:19.119+0000 7f2076a99640 1 -- 172.21.15.84:0/260744131 wait complete. 2024-03-29T10:39:19.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:19.119+0000 7f2076a99640 1 librados: shutdown 2024-03-29T10:39:19.120 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:25.927 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:26.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.098+0000 7f8d7ffff640 10 monclient: get_monmap_and_config 2024-03-29T10:39:26.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.098+0000 7f8d7ffff640 10 monclient: build_initial_monmap 2024-03-29T10:39:26.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.098+0000 7f8d7ffff640 10 monclient: monmap: 2024-03-29T10:39:26.083 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:39:26.083 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:26.083 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:39:26.099813+0000 2024-03-29T10:39:26.083 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:39:26.099813+0000 2024-03-29T10:39:26.083 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:26.083 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:26.084 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:39:26.084 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:39:26.084 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:39:26.084 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:26.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.099+0000 7f8d7ffff640 1 Processor -- start 2024-03-29T10:39:26.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.099+0000 7f8d7ffff640 1 -- start start 2024-03-29T10:39:26.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.099+0000 7f8d7ffff640 10 monclient: init 2024-03-29T10:39:26.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.100+0000 7f8d7ffff640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:26.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.100+0000 7f8d7ffff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:39:26.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.100+0000 7f8d7ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f8d801433e0 addr v1:172.21.15.84:6789/0 2024-03-29T10:39:26.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.100+0000 7f8d7ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f8d800abbf0 addr v1:172.21.15.84:6790/0 2024-03-29T10:39:26.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.100+0000 7f8d7ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f8d80143e00 addr v1:172.21.15.86:6789/0 2024-03-29T10:39:26.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.100+0000 7f8d7ffff640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d800adc80 con 0x7f8d801433e0 2024-03-29T10:39:26.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.100+0000 7f8d7ffff640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d800aded0 con 0x7f8d80143e00 2024-03-29T10:39:26.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.100+0000 7f8d7ffff640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d800ac110 con 0x7f8d800abbf0 2024-03-29T10:39:26.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.100+0000 7f8d7ffff640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:26.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.100+0000 7f8d7ffff640 10 monclient(hunting): authenticate will time out at 1068.363790s 2024-03-29T10:39:26.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.100+0000 7f8d7effd640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f8d801433e0 0x7f8d800ac630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:43216/0 (socket says 172.21.15.84:43216) 2024-03-29T10:39:26.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.100+0000 7f8d7effd640 1 -- 172.21.15.84:0/2367733446 learned_addr learned my addr 172.21.15.84:0/2367733446 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:39:26.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.101+0000 7f8d7dffb640 1 -- 172.21.15.84:0/2367733446 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832849072 0 0) 0x7f8d800ac110 con 0x7f8d800abbf0 2024-03-29T10:39:26.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.101+0000 7f8d7dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:26.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.101+0000 7f8d7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:26.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.101+0000 7f8d7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:26.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.101+0000 7f8d7dffb640 10 monclient(hunting): my global_id is 4421 2024-03-29T10:39:26.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.101+0000 7f8d7dffb640 1 -- 172.21.15.84:0/2367733446 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d68003520 con 0x7f8d800abbf0 2024-03-29T10:39:26.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.101+0000 7f8d7dffb640 1 -- 172.21.15.84:0/2367733446 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131923368 0 0) 0x7f8d800adc80 con 0x7f8d801433e0 2024-03-29T10:39:26.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.101+0000 7f8d7dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:26.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.101+0000 7f8d7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:26.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.101+0000 7f8d7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:26.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.101+0000 7f8d7dffb640 10 monclient(hunting): my global_id is 4404 2024-03-29T10:39:26.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.101+0000 7f8d7dffb640 1 -- 172.21.15.84:0/2367733446 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d68003840 con 0x7f8d801433e0 2024-03-29T10:39:26.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.102+0000 7f8d7dffb640 1 -- 172.21.15.84:0/2367733446 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 134580214 0 0) 0x7f8d68003520 con 0x7f8d800abbf0 2024-03-29T10:39:26.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.102+0000 7f8d7dffb640 1 -- 172.21.15.84:0/2367733446 >> v1:172.21.15.86:6789/0 conn(0x7f8d80143e00 legacy=0x7f8d8014b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:26.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.102+0000 7f8d7dffb640 1 -- 172.21.15.84:0/2367733446 >> v1:172.21.15.84:6789/0 conn(0x7f8d801433e0 legacy=0x7f8d800ac630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:26.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.102+0000 7f8d7dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:39:26.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.102+0000 7f8d7dffb640 1 monclient: found mon.noname-b 2024-03-29T10:39:26.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.102+0000 7f8d7dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:26.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.102+0000 7f8d7dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:39:26.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.102+0000 7f8d7dffb640 1 -- 172.21.15.84:0/2367733446 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d800acd40 con 0x7f8d800abbf0 2024-03-29T10:39:26.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.102+0000 7f8d7dffb640 10 monclient: _finish_auth 0 2024-03-29T10:39:26.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.102+0000 7f8d7dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:39:26.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.102+0000 7f8d7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:26.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.102+0000 7f8d7ffff640 5 monclient: authenticate success, global_id 4421 2024-03-29T10:39:26.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.102+0000 7f8d7ffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:26.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.102+0000 7f8d7dffb640 1 -- 172.21.15.84:0/2367733446 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f8d74001670 con 0x7f8d800abbf0 2024-03-29T10:39:26.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.102+0000 7f8d7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:26.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.102+0000 7f8d7dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:26.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.102+0000 7f8d7dffb640 10 monclient: dump: 2024-03-29T10:39:26.091 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:26.091 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:26.091 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:26.091 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:26.091 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:26.091 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:26.091 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:26.091 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:26.092 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:26.092 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:26.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.102+0000 7f8d7ffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:26.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.102+0000 7f8d7dffb640 1 -- 172.21.15.84:0/2367733446 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d74001850 con 0x7f8d800abbf0 2024-03-29T10:39:26.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.102+0000 7f8d7dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:26.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.102+0000 7f8d7ffff640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:26.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.102+0000 7f8d7ffff640 4 set_mon_vals no callback set 2024-03-29T10:39:26.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.102+0000 7f8d7dffb640 1 -- 172.21.15.84:0/2367733446 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f8d74001670 con 0x7f8d800abbf0 2024-03-29T10:39:26.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.102+0000 7f8d7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:26.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.102+0000 7f8d7dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:26.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.102+0000 7f8d7dffb640 10 monclient: dump: 2024-03-29T10:39:26.092 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:26.092 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:26.092 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:26.092 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:26.093 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:26.093 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:26.093 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:26.093 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:26.093 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:26.094 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:26.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.103+0000 7f8d7ffff640 10 monclient: shutdown 2024-03-29T10:39:26.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.103+0000 7f8d7ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:26.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.103+0000 7f8d7ffff640 1 -- 172.21.15.84:0/2367733446 >> v1:172.21.15.84:6790/0 conn(0x7f8d800abbf0 legacy=0x7f8d80072120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:26.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.103+0000 7f8d7ffff640 1 -- 172.21.15.84:0/2367733446 shutdown_connections 2024-03-29T10:39:26.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.103+0000 7f8d7ffff640 1 -- 172.21.15.84:0/2367733446 >> 172.21.15.84:0/2367733446 conn(0x7f8d800b3c50 msgr2=0x7f8d800b4030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:26.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.103+0000 7f8d7ffff640 1 -- 172.21.15.84:0/2367733446 shutdown_connections 2024-03-29T10:39:26.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.103+0000 7f8d7ffff640 1 -- 172.21.15.84:0/2367733446 wait complete. 2024-03-29T10:39:26.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.104+0000 7f8d7ffff640 10 monclient: build_initial_monmap 2024-03-29T10:39:26.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.104+0000 7f8d7ffff640 10 monclient: monmap: 2024-03-29T10:39:26.099 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:39:26.099 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:26.099 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:39:26.099 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:39:26.099 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:26.100 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:26.100 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:39:26.100 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:39:26.100 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:39:26.100 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:26.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.104+0000 7f8d7ffff640 1 librados: starting msgr at 2024-03-29T10:39:26.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.104+0000 7f8d7ffff640 1 librados: starting objecter 2024-03-29T10:39:26.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.105+0000 7f8d7ffff640 1 Processor -- start 2024-03-29T10:39:26.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.105+0000 7f8d7ffff640 1 -- start start 2024-03-29T10:39:26.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.105+0000 7f8d7ffff640 1 librados: setting wanted keys 2024-03-29T10:39:26.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.105+0000 7f8d7ffff640 1 librados: calling monclient init 2024-03-29T10:39:26.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.105+0000 7f8d7ffff640 10 monclient: init 2024-03-29T10:39:26.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.106+0000 7f8d7ffff640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:26.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.106+0000 7f8d7ffff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:39:26.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.106+0000 7f8d7ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f8d800abbf0 addr v1:172.21.15.84:6789/0 2024-03-29T10:39:26.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.106+0000 7f8d7ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f8d801433e0 addr v1:172.21.15.86:6789/0 2024-03-29T10:39:26.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.106+0000 7f8d7ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f8d80143e00 addr v1:172.21.15.84:6790/0 2024-03-29T10:39:26.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.106+0000 7f8d7ffff640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d8013fad0 con 0x7f8d800abbf0 2024-03-29T10:39:26.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.106+0000 7f8d7ffff640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d801dc270 con 0x7f8d801433e0 2024-03-29T10:39:26.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.106+0000 7f8d7ffff640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d8013d9f0 con 0x7f8d80143e00 2024-03-29T10:39:26.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.106+0000 7f8d7ffff640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:26.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.106+0000 7f8d7ffff640 10 monclient(hunting): authenticate will time out at 1068.369699s 2024-03-29T10:39:26.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.106+0000 7f8d7e7fc640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f8d801433e0 0x7f8d801db450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:57772/0 (socket says 172.21.15.84:57772) 2024-03-29T10:39:26.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.106+0000 7f8d7e7fc640 1 -- 172.21.15.84:0/3824095190 learned_addr learned my addr 172.21.15.84:0/3824095190 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:39:26.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.107+0000 7f8d677fe640 10 client.?.objecter ms_handle_connect 0x7f8d800abbf0 2024-03-29T10:39:26.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.107+0000 7f8d677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:26.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.107+0000 7f8d677fe640 10 client.?.objecter ms_handle_connect 0x7f8d80143e00 2024-03-29T10:39:26.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.107+0000 7f8d677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:26.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.107+0000 7f8d677fe640 1 -- 172.21.15.84:0/3824095190 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 976431223 0 0) 0x7f8d8013fad0 con 0x7f8d800abbf0 2024-03-29T10:39:26.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.107+0000 7f8d677fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:26.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.107+0000 7f8d677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:26.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.107+0000 7f8d677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:26.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.107+0000 7f8d677fe640 10 monclient(hunting): my global_id is 4407 2024-03-29T10:39:26.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.107+0000 7f8d677fe640 1 -- 172.21.15.84:0/3824095190 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d6c004eb0 con 0x7f8d800abbf0 2024-03-29T10:39:26.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.107+0000 7f8d677fe640 1 -- 172.21.15.84:0/3824095190 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3200260586 0 0) 0x7f8d8013d9f0 con 0x7f8d80143e00 2024-03-29T10:39:26.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.107+0000 7f8d677fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:26.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.107+0000 7f8d677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:26.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.107+0000 7f8d677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:26.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.107+0000 7f8d677fe640 10 monclient(hunting): my global_id is 4424 2024-03-29T10:39:26.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.107+0000 7f8d677fe640 1 -- 172.21.15.84:0/3824095190 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d8013fad0 con 0x7f8d80143e00 2024-03-29T10:39:26.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.107+0000 7f8d677fe640 1 -- 172.21.15.84:0/3824095190 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 919734679 0 0) 0x7f8d6c004eb0 con 0x7f8d800abbf0 2024-03-29T10:39:26.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.107+0000 7f8d677fe640 1 -- 172.21.15.84:0/3824095190 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d8013d9f0 con 0x7f8d800abbf0 2024-03-29T10:39:26.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d677fe640 1 -- 172.21.15.84:0/3824095190 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f8d70001930 con 0x7f8d800abbf0 2024-03-29T10:39:26.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:26.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d677fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:39:26.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d677fe640 10 monclient(hunting): dump: 2024-03-29T10:39:26.103 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:26.103 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:26.103 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:26.103 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:26.103 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:26.103 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:26.103 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:26.103 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:26.103 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:26.103 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:26.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d677fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:26.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d677fe640 1 -- 172.21.15.84:0/3824095190 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 305038641 0 0) 0x7f8d8013fad0 con 0x7f8d80143e00 2024-03-29T10:39:26.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d677fe640 1 -- 172.21.15.84:0/3824095190 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d6c004eb0 con 0x7f8d80143e00 2024-03-29T10:39:26.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d677fe640 10 client.?.objecter ms_handle_connect 0x7f8d801433e0 2024-03-29T10:39:26.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:26.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d677fe640 1 -- 172.21.15.84:0/3824095190 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f8d78001b00 con 0x7f8d80143e00 2024-03-29T10:39:26.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:26.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d677fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:26.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d677fe640 10 monclient(hunting): dump: 2024-03-29T10:39:26.104 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:26.104 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:26.104 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:26.104 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:26.104 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:26.104 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:26.104 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:26.104 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:26.105 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:26.105 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:26.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d677fe640 1 -- 172.21.15.84:0/3824095190 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 662553494 0 0) 0x7f8d8013d9f0 con 0x7f8d800abbf0 2024-03-29T10:39:26.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d677fe640 1 -- 172.21.15.84:0/3824095190 >> v1:172.21.15.84:6790/0 conn(0x7f8d80143e00 legacy=0x7f8d801dbb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:26.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d677fe640 1 -- 172.21.15.84:0/3824095190 >> v1:172.21.15.86:6789/0 conn(0x7f8d801433e0 legacy=0x7f8d801db450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:26.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d677fe640 10 monclient: _finish_hunting 0 2024-03-29T10:39:26.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d677fe640 1 monclient: found mon.a 2024-03-29T10:39:26.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:26.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d677fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:39:26.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d677fe640 1 -- 172.21.15.84:0/3824095190 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d8013db80 con 0x7f8d800abbf0 2024-03-29T10:39:26.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d677fe640 10 monclient: _finish_auth 0 2024-03-29T10:39:26.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d677fe640 10 monclient: _check_auth_tickets 2024-03-29T10:39:26.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:26.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d7ffff640 5 monclient: authenticate success, global_id 4407 2024-03-29T10:39:26.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d7ffff640 10 monclient: _renew_subs 2024-03-29T10:39:26.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d7ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:39:26.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d7ffff640 1 -- 172.21.15.84:0/3824095190 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d8013ded0 con 0x7f8d800abbf0 2024-03-29T10:39:26.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d7ffff640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:26.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d7ffff640 10 monclient: _renew_subs 2024-03-29T10:39:26.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d7ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:39:26.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d7ffff640 1 -- 172.21.15.84:0/3824095190 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d801401b0 con 0x7f8d800abbf0 2024-03-29T10:39:26.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.108+0000 7f8d7ffff640 1 librados: init done 2024-03-29T10:39:26.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.109+0000 7f8d677fe640 1 -- 172.21.15.84:0/3824095190 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d700047e0 con 0x7f8d800abbf0 2024-03-29T10:39:26.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.109+0000 7f8d677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:26.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.109+0000 7f8d7cff9640 4 set_mon_vals no callback set 2024-03-29T10:39:26.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.110+0000 7f8d677fe640 1 -- 172.21.15.84:0/3824095190 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f8d70004af0 con 0x7f8d800abbf0 2024-03-29T10:39:26.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.110+0000 7f8d677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:26.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.110+0000 7f8d677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:39:26.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.110+0000 7f8d677fe640 10 monclient: dump: 2024-03-29T10:39:26.107 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:26.107 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:26.107 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:26.107 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:26.107 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:26.107 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:26.107 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:26.107 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:26.107 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:26.107 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:26.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.110+0000 7f8d7ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:26.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.110+0000 7f8d7ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:26.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.110+0000 7f8d7ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:39:26.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.110+0000 7f8d7ffff640 1 -- 172.21.15.84:0/3824095190 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d60005080 con 0x7f8d800abbf0 2024-03-29T10:39:26.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.111+0000 7f8d677fe640 1 -- 172.21.15.84:0/3824095190 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f8d7001aac0 con 0x7f8d800abbf0 2024-03-29T10:39:26.107 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.112+0000 7f8d677fe640 1 -- 172.21.15.84:0/3824095190 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f8d70084a00 con 0x7f8d800abbf0 2024-03-29T10:39:26.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.112+0000 7f8d677fe640 10 client.4407.objecter ms_dispatch 0x7f8d80000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:26.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.112+0000 7f8d677fe640 3 client.4407.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:26.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.112+0000 7f8d677fe640 3 client.4407.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:26.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.112+0000 7f8d677fe640 20 client.4407.objecter dump_active .. 0 homeless 2024-03-29T10:39:26.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.113+0000 7f8d7e7fc640 10 monclient: get_auth_request con 0x7f8d6c06de80 auth_method 0 2024-03-29T10:39:26.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.113+0000 7f8d677fe640 10 client.4407.objecter ms_handle_connect 0x7f8d6c06de80 2024-03-29T10:39:26.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.125+0000 7f8d677fe640 1 -- 172.21.15.84:0/3824095190 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8d700502e0 con 0x7f8d800abbf0 2024-03-29T10:39:26.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.125+0000 7f8d677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:26.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.125+0000 7f8d677fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:26.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.485+0000 7f8d7ffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:26.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.485+0000 7f8d7ffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:26.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.485+0000 7f8d7ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:39:26.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.485+0000 7f8d7ffff640 1 -- 172.21.15.84:0/3824095190 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d60005330 con 0x7f8d800abbf0 2024-03-29T10:39:26.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.486+0000 7f8d677fe640 1 -- 172.21.15.84:0/3824095190 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f8d70052210 con 0x7f8d800abbf0 2024-03-29T10:39:26.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.486+0000 7f8d677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:26.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.486+0000 7f8d677fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:26.470 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:39:26.470 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:26.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.489+0000 7f8d7ffff640 10 librados: watch_flush enter 2024-03-29T10:39:26.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.489+0000 7f8d7ffff640 10 librados: watch_flush exit 2024-03-29T10:39:26.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.489+0000 7f8d7ffff640 20 client.4407.objecter shutdown clearing up homeless session... 2024-03-29T10:39:26.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.490+0000 7f8d7ffff640 10 client.4407.objecter successfully canceled tick 2024-03-29T10:39:26.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.490+0000 7f8d7ffff640 1 -- 172.21.15.84:0/3824095190 >> v1:172.21.15.86:6816/3337225337 conn(0x7f8d6c06de80 legacy=0x7f8d6c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:26.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.490+0000 7f8d7ffff640 10 monclient: shutdown 2024-03-29T10:39:26.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.490+0000 7f8d7ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:26.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.490+0000 7f8d7ffff640 1 -- 172.21.15.84:0/3824095190 >> v1:172.21.15.84:6789/0 conn(0x7f8d800abbf0 legacy=0x7f8d8013cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:26.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.490+0000 7f8d7ffff640 1 -- 172.21.15.84:0/3824095190 shutdown_connections 2024-03-29T10:39:26.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.490+0000 7f8d7ffff640 1 -- 172.21.15.84:0/3824095190 >> 172.21.15.84:0/3824095190 conn(0x7f8d800b3c50 msgr2=0x7f8d800b2b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:26.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.490+0000 7f8d7ffff640 1 -- 172.21.15.84:0/3824095190 shutdown_connections 2024-03-29T10:39:26.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.490+0000 7f8d7ffff640 1 -- 172.21.15.84:0/3824095190 wait complete. 2024-03-29T10:39:26.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:26.490+0000 7f8d7ffff640 1 librados: shutdown 2024-03-29T10:39:26.488 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:33.295 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:33.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.442+0000 7fa1d92c0640 10 monclient: get_monmap_and_config 2024-03-29T10:39:33.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.442+0000 7fa1d92c0640 10 monclient: build_initial_monmap 2024-03-29T10:39:33.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.442+0000 7fa1d92c0640 10 monclient: monmap: 2024-03-29T10:39:33.426 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:39:33.426 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:33.426 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:39:33.443085+0000 2024-03-29T10:39:33.426 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:39:33.443085+0000 2024-03-29T10:39:33.426 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:33.427 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:33.427 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:39:33.427 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:39:33.427 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:39:33.427 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:33.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.443+0000 7fa1d92c0640 1 Processor -- start 2024-03-29T10:39:33.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.443+0000 7fa1d92c0640 1 -- start start 2024-03-29T10:39:33.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.443+0000 7fa1d92c0640 10 monclient: init 2024-03-29T10:39:33.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.443+0000 7fa1d92c0640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:33.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.443+0000 7fa1d92c0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:39:33.428 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.443+0000 7fa1d92c0640 10 monclient(hunting): picked mon.noname-b con 0x7fa1d41457f0 addr v1:172.21.15.84:6790/0 2024-03-29T10:39:33.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.443+0000 7fa1d92c0640 10 monclient(hunting): picked mon.noname-c con 0x7fa1d413f650 addr v1:172.21.15.86:6789/0 2024-03-29T10:39:33.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.443+0000 7fa1d92c0640 10 monclient(hunting): picked mon.noname-a con 0x7fa1d40a9d70 addr v1:172.21.15.84:6789/0 2024-03-29T10:39:33.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.443+0000 7fa1d92c0640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d40aaef0 con 0x7fa1d40a9d70 2024-03-29T10:39:33.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.443+0000 7fa1d92c0640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d4145bd0 con 0x7fa1d413f650 2024-03-29T10:39:33.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.443+0000 7fa1d92c0640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d4145eb0 con 0x7fa1d41457f0 2024-03-29T10:39:33.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.443+0000 7fa1d92c0640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:33.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.443+0000 7fa1d92c0640 10 monclient(hunting): authenticate will time out at 1075.706971s 2024-03-29T10:39:33.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.443+0000 7fa1d2ffd640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fa1d41457f0 0x7fa1d413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:37010/0 (socket says 172.21.15.84:37010) 2024-03-29T10:39:33.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.444+0000 7fa1d2ffd640 1 -- 172.21.15.84:0/1059255117 learned_addr learned my addr 172.21.15.84:0/1059255117 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:39:33.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.444+0000 7fa1d1ffb640 1 -- 172.21.15.84:0/1059255117 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2753315692 0 0) 0x7fa1d4145eb0 con 0x7fa1d41457f0 2024-03-29T10:39:33.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.444+0000 7fa1d1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:33.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.444+0000 7fa1d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:33.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.444+0000 7fa1d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:33.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.444+0000 7fa1d1ffb640 10 monclient(hunting): my global_id is 4427 2024-03-29T10:39:33.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.444+0000 7fa1d1ffb640 1 -- 172.21.15.84:0/1059255117 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1bc003520 con 0x7fa1d41457f0 2024-03-29T10:39:33.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.444+0000 7fa1d1ffb640 1 -- 172.21.15.84:0/1059255117 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 664744770 0 0) 0x7fa1bc003520 con 0x7fa1d41457f0 2024-03-29T10:39:33.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.444+0000 7fa1d1ffb640 1 -- 172.21.15.84:0/1059255117 >> v1:172.21.15.86:6789/0 conn(0x7fa1d413f650 legacy=0x7fa1d40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:39:33.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.444+0000 7fa1d1ffb640 1 -- 172.21.15.84:0/1059255117 >> v1:172.21.15.84:6789/0 conn(0x7fa1d40a9d70 legacy=0x7fa1d414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:39:33.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.445+0000 7fa1d1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:39:33.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.445+0000 7fa1d1ffb640 1 monclient: found mon.noname-b 2024-03-29T10:39:33.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.445+0000 7fa1d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:33.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.445+0000 7fa1d1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:39:33.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.445+0000 7fa1d1ffb640 1 -- 172.21.15.84:0/1059255117 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d4146340 con 0x7fa1d41457f0 2024-03-29T10:39:33.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.445+0000 7fa1d1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:39:33.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.445+0000 7fa1d1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:39:33.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.445+0000 7fa1d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:33.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.445+0000 7fa1d1ffb640 1 -- 172.21.15.84:0/1059255117 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa1cc0019f0 con 0x7fa1d41457f0 2024-03-29T10:39:33.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.445+0000 7fa1d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:33.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.445+0000 7fa1d1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:33.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.445+0000 7fa1d1ffb640 10 monclient: dump: 2024-03-29T10:39:33.434 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:33.434 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:33.434 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:33.434 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:33.434 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:33.434 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:33.434 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:33.434 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:33.434 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:33.434 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:33.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.445+0000 7fa1d92c0640 5 monclient: authenticate success, global_id 4427 2024-03-29T10:39:33.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.445+0000 7fa1d92c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:33.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.446+0000 7fa1d1ffb640 1 -- 172.21.15.84:0/1059255117 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1cc001c50 con 0x7fa1d41457f0 2024-03-29T10:39:33.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.446+0000 7fa1d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:33.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.446+0000 7fa1d92c0640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:33.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.446+0000 7fa1d92c0640 4 set_mon_vals no callback set 2024-03-29T10:39:33.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.446+0000 7fa1d1ffb640 1 -- 172.21.15.84:0/1059255117 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa1cc0020a0 con 0x7fa1d41457f0 2024-03-29T10:39:33.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.446+0000 7fa1d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:33.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.446+0000 7fa1d1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:33.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.446+0000 7fa1d1ffb640 10 monclient: dump: 2024-03-29T10:39:33.435 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:33.435 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:33.435 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:33.435 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:33.435 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:33.435 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:33.435 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:33.435 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:33.436 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:33.436 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:33.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.447+0000 7fa1d92c0640 10 monclient: shutdown 2024-03-29T10:39:33.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.447+0000 7fa1d92c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:33.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.447+0000 7fa1d92c0640 1 -- 172.21.15.84:0/1059255117 >> v1:172.21.15.84:6790/0 conn(0x7fa1d41457f0 legacy=0x7fa1d413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:33.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.447+0000 7fa1d92c0640 1 -- 172.21.15.84:0/1059255117 shutdown_connections 2024-03-29T10:39:33.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.447+0000 7fa1d92c0640 1 -- 172.21.15.84:0/1059255117 >> 172.21.15.84:0/1059255117 conn(0x7fa1d4066a00 msgr2=0x7fa1d4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:33.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.447+0000 7fa1d92c0640 1 -- 172.21.15.84:0/1059255117 shutdown_connections 2024-03-29T10:39:33.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.447+0000 7fa1d92c0640 1 -- 172.21.15.84:0/1059255117 wait complete. 2024-03-29T10:39:33.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.448+0000 7fa1d92c0640 10 monclient: build_initial_monmap 2024-03-29T10:39:33.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.448+0000 7fa1d92c0640 10 monclient: monmap: 2024-03-29T10:39:33.436 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:39:33.436 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:33.436 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:39:33.437 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:39:33.437 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:33.437 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:33.437 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:39:33.437 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:39:33.437 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:39:33.437 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:33.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.448+0000 7fa1d92c0640 1 librados: starting msgr at 2024-03-29T10:39:33.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.448+0000 7fa1d92c0640 1 librados: starting objecter 2024-03-29T10:39:33.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.449+0000 7fa1d92c0640 1 Processor -- start 2024-03-29T10:39:33.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.449+0000 7fa1d92c0640 1 -- start start 2024-03-29T10:39:33.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.449+0000 7fa1d92c0640 1 librados: setting wanted keys 2024-03-29T10:39:33.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.449+0000 7fa1d92c0640 1 librados: calling monclient init 2024-03-29T10:39:33.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.449+0000 7fa1d92c0640 10 monclient: init 2024-03-29T10:39:33.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.449+0000 7fa1d92c0640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:33.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.449+0000 7fa1d92c0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:39:33.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.449+0000 7fa1d92c0640 10 monclient(hunting): picked mon.noname-b con 0x7fa1d40a9d70 addr v1:172.21.15.86:6789/0 2024-03-29T10:39:33.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.449+0000 7fa1d92c0640 10 monclient(hunting): picked mon.noname-c con 0x7fa1d413f650 addr v1:172.21.15.84:6790/0 2024-03-29T10:39:33.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.450+0000 7fa1d27fc640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fa1d413f650 0x7fa1d4135370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:37110/0 (socket says 172.21.15.84:37110) 2024-03-29T10:39:33.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.450+0000 7fa1d2ffd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa1d40a9d70 0x7fa1d4137140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:50480/0 (socket says 172.21.15.84:50480) 2024-03-29T10:39:33.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.450+0000 7fa1d27fc640 1 -- 172.21.15.84:0/835474457 learned_addr learned my addr 172.21.15.84:0/835474457 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:39:33.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.450+0000 7fa1d92c0640 10 monclient(hunting): picked mon.noname-a con 0x7fa1d41457f0 addr v1:172.21.15.84:6789/0 2024-03-29T10:39:33.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.450+0000 7fa1d92c0640 1 -- 172.21.15.84:0/835474457 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d4136190 con 0x7fa1d41457f0 2024-03-29T10:39:33.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.451+0000 7fa1d92c0640 1 -- 172.21.15.84:0/835474457 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d4071b70 con 0x7fa1d40a9d70 2024-03-29T10:39:33.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.451+0000 7fa1d92c0640 1 -- 172.21.15.84:0/835474457 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d4071e50 con 0x7fa1d413f650 2024-03-29T10:39:33.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.450+0000 7fa1bb7fe640 10 client.?.objecter ms_handle_connect 0x7fa1d413f650 2024-03-29T10:39:33.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.451+0000 7fa1bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:33.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.451+0000 7fa1bb7fe640 10 client.?.objecter ms_handle_connect 0x7fa1d40a9d70 2024-03-29T10:39:33.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.451+0000 7fa1bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:33.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.451+0000 7fa1bb7fe640 1 -- 172.21.15.84:0/835474457 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1871066353 0 0) 0x7fa1d4071e50 con 0x7fa1d413f650 2024-03-29T10:39:33.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.451+0000 7fa1d92c0640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:33.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.451+0000 7fa1d92c0640 10 monclient(hunting): authenticate will time out at 1075.714620s 2024-03-29T10:39:33.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.451+0000 7fa1bb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:33.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.451+0000 7fa1bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:33.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.451+0000 7fa1bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:33.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.451+0000 7fa1bb7fe640 10 monclient(hunting): my global_id is 4430 2024-03-29T10:39:33.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.451+0000 7fa1bb7fe640 1 -- 172.21.15.84:0/835474457 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1c0004eb0 con 0x7fa1d413f650 2024-03-29T10:39:33.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.451+0000 7fa1bb7fe640 1 -- 172.21.15.84:0/835474457 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2944581650 0 0) 0x7fa1d4071b70 con 0x7fa1d40a9d70 2024-03-29T10:39:33.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.451+0000 7fa1bb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:33.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.451+0000 7fa1bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:33.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.451+0000 7fa1bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:33.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.451+0000 7fa1bb7fe640 10 monclient(hunting): my global_id is 4378 2024-03-29T10:39:33.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.451+0000 7fa1bb7fe640 1 -- 172.21.15.84:0/835474457 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1d4071e50 con 0x7fa1d40a9d70 2024-03-29T10:39:33.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.452+0000 7fa1bb7fe640 1 -- 172.21.15.84:0/835474457 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3948354964 0 0) 0x7fa1c0004eb0 con 0x7fa1d413f650 2024-03-29T10:39:33.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.452+0000 7fa1bb7fe640 1 -- 172.21.15.84:0/835474457 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1d4071b70 con 0x7fa1d413f650 2024-03-29T10:39:33.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.452+0000 7fa1bb7fe640 1 -- 172.21.15.84:0/835474457 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa1c4001960 con 0x7fa1d413f650 2024-03-29T10:39:33.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.452+0000 7fa1bb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:33.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.452+0000 7fa1bb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:33.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.452+0000 7fa1bb7fe640 10 monclient(hunting): dump: 2024-03-29T10:39:33.447 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:33.447 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:33.447 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:33.447 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:33.447 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:33.447 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:33.447 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:33.447 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:33.447 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:33.447 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:33.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.452+0000 7fa1bb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:33.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.452+0000 7fa1bb7fe640 1 -- 172.21.15.84:0/835474457 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3722584801 0 0) 0x7fa1d4071e50 con 0x7fa1d40a9d70 2024-03-29T10:39:33.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.452+0000 7fa1bb7fe640 1 -- 172.21.15.84:0/835474457 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1c0004eb0 con 0x7fa1d40a9d70 2024-03-29T10:39:33.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.452+0000 7fa1bb7fe640 1 -- 172.21.15.84:0/835474457 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa1cc0020a0 con 0x7fa1d40a9d70 2024-03-29T10:39:33.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.452+0000 7fa1bb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:33.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.452+0000 7fa1bb7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:33.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.452+0000 7fa1bb7fe640 10 monclient(hunting): dump: 2024-03-29T10:39:33.448 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:33.448 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:33.448 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:33.448 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:33.449 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:33.449 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:33.449 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:33.449 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:33.449 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:33.449 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:33.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.452+0000 7fa1bb7fe640 1 -- 172.21.15.84:0/835474457 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1256050934 0 0) 0x7fa1d4071b70 con 0x7fa1d413f650 2024-03-29T10:39:33.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.452+0000 7fa1bb7fe640 1 -- 172.21.15.84:0/835474457 >> v1:172.21.15.86:6789/0 conn(0x7fa1d40a9d70 legacy=0x7fa1d4137140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:33.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.452+0000 7fa1bb7fe640 1 -- 172.21.15.84:0/835474457 >> v1:172.21.15.84:6789/0 conn(0x7fa1d41457f0 legacy=0x7fa1d4135a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:39:33.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.453+0000 7fa1bb7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:39:33.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.453+0000 7fa1bb7fe640 1 monclient: found mon.c 2024-03-29T10:39:33.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.453+0000 7fa1bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:33.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.453+0000 7fa1bb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:39:33.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.453+0000 7fa1bb7fe640 1 -- 172.21.15.84:0/835474457 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d4136d20 con 0x7fa1d413f650 2024-03-29T10:39:33.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.453+0000 7fa1bb7fe640 10 monclient: _finish_auth 0 2024-03-29T10:39:33.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.453+0000 7fa1bb7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:39:33.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.453+0000 7fa1bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:33.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.453+0000 7fa1d92c0640 5 monclient: authenticate success, global_id 4430 2024-03-29T10:39:33.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.453+0000 7fa1d92c0640 10 monclient: _renew_subs 2024-03-29T10:39:33.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.453+0000 7fa1d92c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:39:33.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.453+0000 7fa1d92c0640 1 -- 172.21.15.84:0/835474457 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1d40711d0 con 0x7fa1d413f650 2024-03-29T10:39:33.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.453+0000 7fa1d92c0640 10 client.4430.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:33.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.453+0000 7fa1d92c0640 10 monclient: _renew_subs 2024-03-29T10:39:33.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.453+0000 7fa1d92c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:39:33.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.453+0000 7fa1d92c0640 1 -- 172.21.15.84:0/835474457 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1d4136410 con 0x7fa1d413f650 2024-03-29T10:39:33.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.453+0000 7fa1d92c0640 1 librados: init done 2024-03-29T10:39:33.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.454+0000 7fa1bb7fe640 1 -- 172.21.15.84:0/835474457 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1c40035f0 con 0x7fa1d413f650 2024-03-29T10:39:33.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.454+0000 7fa1bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:33.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.454+0000 7fa1d17fa640 4 set_mon_vals no callback set 2024-03-29T10:39:33.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.454+0000 7fa1bb7fe640 1 -- 172.21.15.84:0/835474457 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa1c4003940 con 0x7fa1d413f650 2024-03-29T10:39:33.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.454+0000 7fa1bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:33.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.454+0000 7fa1bb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:33.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.454+0000 7fa1bb7fe640 10 monclient: dump: 2024-03-29T10:39:33.451 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:33.451 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:33.451 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:33.452 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:33.452 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:33.452 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:33.452 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:33.452 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:33.452 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:33.452 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:33.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.455+0000 7fa1d92c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:33.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.456+0000 7fa1d92c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:33.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.456+0000 7fa1d92c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:39:33.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.456+0000 7fa1d92c0640 1 -- 172.21.15.84:0/835474457 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1b4005080 con 0x7fa1d413f650 2024-03-29T10:39:33.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.456+0000 7fa1bb7fe640 1 -- 172.21.15.84:0/835474457 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7fa1c4003aa0 con 0x7fa1d413f650 2024-03-29T10:39:33.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.458+0000 7fa1bb7fe640 1 -- 172.21.15.84:0/835474457 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7fa1c4084a30 con 0x7fa1d413f650 2024-03-29T10:39:33.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.458+0000 7fa1bb7fe640 10 client.4430.objecter ms_dispatch 0x7fa1d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:33.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.458+0000 7fa1bb7fe640 3 client.4430.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:33.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.458+0000 7fa1bb7fe640 3 client.4430.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:33.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.458+0000 7fa1bb7fe640 20 client.4430.objecter dump_active .. 0 homeless 2024-03-29T10:39:33.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.458+0000 7fa1d2ffd640 10 monclient: get_auth_request con 0x7fa1c006de30 auth_method 0 2024-03-29T10:39:33.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.459+0000 7fa1bb7fe640 10 client.4430.objecter ms_handle_connect 0x7fa1c006de30 2024-03-29T10:39:33.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.464+0000 7fa1bb7fe640 1 -- 172.21.15.84:0/835474457 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa1c4050310 con 0x7fa1d413f650 2024-03-29T10:39:33.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.464+0000 7fa1bb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:33.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.464+0000 7fa1bb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:33.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.768+0000 7fa1d92c0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:33.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.768+0000 7fa1d92c0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:33.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.768+0000 7fa1d92c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:39:33.752 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.768+0000 7fa1d92c0640 1 -- 172.21.15.84:0/835474457 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1b4005330 con 0x7fa1d413f650 2024-03-29T10:39:33.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.768+0000 7fa1bb7fe640 1 -- 172.21.15.84:0/835474457 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7fa1c4052240 con 0x7fa1d413f650 2024-03-29T10:39:33.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.768+0000 7fa1bb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:33.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.768+0000 7fa1bb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:33.753 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:39:33.753 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:33.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.772+0000 7fa1d92c0640 10 librados: watch_flush enter 2024-03-29T10:39:33.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.772+0000 7fa1d92c0640 10 librados: watch_flush exit 2024-03-29T10:39:33.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.772+0000 7fa1d92c0640 20 client.4430.objecter shutdown clearing up homeless session... 2024-03-29T10:39:33.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.772+0000 7fa1d92c0640 10 client.4430.objecter successfully canceled tick 2024-03-29T10:39:33.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.772+0000 7fa1d92c0640 1 -- 172.21.15.84:0/835474457 >> v1:172.21.15.86:6816/3337225337 conn(0x7fa1c006de30 legacy=0x7fa1c00702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:33.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.772+0000 7fa1d92c0640 10 monclient: shutdown 2024-03-29T10:39:33.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.772+0000 7fa1d92c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:33.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.772+0000 7fa1d92c0640 1 -- 172.21.15.84:0/835474457 >> v1:172.21.15.84:6790/0 conn(0x7fa1d413f650 legacy=0x7fa1d4135370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:33.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.772+0000 7fa1d92c0640 1 -- 172.21.15.84:0/835474457 shutdown_connections 2024-03-29T10:39:33.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.772+0000 7fa1d92c0640 1 -- 172.21.15.84:0/835474457 >> 172.21.15.84:0/835474457 conn(0x7fa1d4066a00 msgr2=0x7fa1d4144b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:33.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.772+0000 7fa1d92c0640 1 -- 172.21.15.84:0/835474457 shutdown_connections 2024-03-29T10:39:33.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.772+0000 7fa1d92c0640 1 -- 172.21.15.84:0/835474457 wait complete. 2024-03-29T10:39:33.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:33.772+0000 7fa1d92c0640 1 librados: shutdown 2024-03-29T10:39:33.771 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:40.579 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:40.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.744+0000 7f48c1f0b640 10 monclient: get_monmap_and_config 2024-03-29T10:39:40.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.744+0000 7f48c1f0b640 10 monclient: build_initial_monmap 2024-03-29T10:39:40.729 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.744+0000 7f48c1f0b640 10 monclient: monmap: 2024-03-29T10:39:40.729 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:39:40.729 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:40.729 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:39:40.746185+0000 2024-03-29T10:39:40.729 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:39:40.746185+0000 2024-03-29T10:39:40.729 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:40.730 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:40.730 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:39:40.730 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:39:40.730 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:39:40.730 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:40.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.747+0000 7f48c1f0b640 1 Processor -- start 2024-03-29T10:39:40.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.747+0000 7f48c1f0b640 1 -- start start 2024-03-29T10:39:40.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.747+0000 7f48c1f0b640 10 monclient: init 2024-03-29T10:39:40.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.747+0000 7f48c1f0b640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:40.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.747+0000 7f48c1f0b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:39:40.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.747+0000 7f48c1f0b640 10 monclient(hunting): picked mon.noname-b con 0x7f48b4006800 addr v1:172.21.15.84:6790/0 2024-03-29T10:39:40.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.747+0000 7f48c1f0b640 10 monclient(hunting): picked mon.noname-c con 0x7f48b4006be0 addr v1:172.21.15.86:6789/0 2024-03-29T10:39:40.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.747+0000 7f48c1f0b640 10 monclient(hunting): picked mon.noname-a con 0x7f48b4031b60 addr v1:172.21.15.84:6789/0 2024-03-29T10:39:40.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.747+0000 7f48c1f0b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48b4029370 con 0x7f48b4031b60 2024-03-29T10:39:40.733 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.747+0000 7f48c1f0b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48b40295e0 con 0x7f48b4006be0 2024-03-29T10:39:40.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.747+0000 7f48c1f0b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48b4031360 con 0x7f48b4006800 2024-03-29T10:39:40.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.747+0000 7f48c1f0b640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:40.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.747+0000 7f48c1f0b640 10 monclient(hunting): authenticate will time out at 1083.012038s 2024-03-29T10:39:40.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.748+0000 7f48c0f09640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f48b4006800 0x7f48b40cad20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:36346/0 (socket says 172.21.15.84:36346) 2024-03-29T10:39:40.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.748+0000 7f48c0f09640 1 -- 172.21.15.84:0/1625798800 learned_addr learned my addr 172.21.15.84:0/1625798800 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:39:40.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.748+0000 7f48bb7fe640 1 -- 172.21.15.84:0/1625798800 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455135609 0 0) 0x7f48b4031360 con 0x7f48b4006800 2024-03-29T10:39:40.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.748+0000 7f48bb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:39:40.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.748+0000 7f48bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:40.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.748+0000 7f48bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:40.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.748+0000 7f48bb7fe640 10 monclient(hunting): my global_id is 4433 2024-03-29T10:39:40.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.748+0000 7f48bb7fe640 1 -- 172.21.15.84:0/1625798800 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48ac003520 con 0x7f48b4006800 2024-03-29T10:39:40.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.748+0000 7f48bb7fe640 1 -- 172.21.15.84:0/1625798800 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1026912878 0 0) 0x7f48ac003520 con 0x7f48b4006800 2024-03-29T10:39:40.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.748+0000 7f48bb7fe640 1 -- 172.21.15.84:0/1625798800 >> v1:172.21.15.86:6789/0 conn(0x7f48b4006be0 legacy=0x7f48b40c4ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:39:40.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.751+0000 7f48bb7fe640 1 -- 172.21.15.84:0/1625798800 >> v1:172.21.15.84:6789/0 conn(0x7f48b4031b60 legacy=0x7f48b40d30c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:39:40.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.751+0000 7f48bb7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:39:40.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.751+0000 7f48bb7fe640 1 monclient: found mon.noname-b 2024-03-29T10:39:40.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.751+0000 7f48bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:40.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.751+0000 7f48bb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:39:40.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.751+0000 7f48bb7fe640 1 -- 172.21.15.84:0/1625798800 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b4031640 con 0x7f48b4006800 2024-03-29T10:39:40.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.751+0000 7f48bb7fe640 10 monclient: _finish_auth 0 2024-03-29T10:39:40.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.751+0000 7f48bb7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:39:40.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.751+0000 7f48bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:40.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.751+0000 7f48bb7fe640 1 -- 172.21.15.84:0/1625798800 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f48a8001670 con 0x7f48b4006800 2024-03-29T10:39:40.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.751+0000 7f48bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:40.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.751+0000 7f48bb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:40.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.751+0000 7f48bb7fe640 10 monclient: dump: 2024-03-29T10:39:40.737 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:40.738 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:40.738 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:40.738 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:40.738 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:40.738 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:40.738 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:40.738 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:40.738 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:40.738 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:40.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.751+0000 7f48c1f0b640 5 monclient: authenticate success, global_id 4433 2024-03-29T10:39:40.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.751+0000 7f48c1f0b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:40.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.751+0000 7f48bb7fe640 1 -- 172.21.15.84:0/1625798800 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48a80018d0 con 0x7f48b4006800 2024-03-29T10:39:40.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.751+0000 7f48bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:40.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.751+0000 7f48bb7fe640 1 -- 172.21.15.84:0/1625798800 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f48a8001ca0 con 0x7f48b4006800 2024-03-29T10:39:40.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.751+0000 7f48bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:40.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.751+0000 7f48bb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:40.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.751+0000 7f48bb7fe640 10 monclient: dump: 2024-03-29T10:39:40.739 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:40.739 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:40.739 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:40.739 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:40.739 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:40.739 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:40.739 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:40.739 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:40.739 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:40.739 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:40.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.751+0000 7f48c1f0b640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:40.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.751+0000 7f48c1f0b640 4 set_mon_vals no callback set 2024-03-29T10:39:40.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.752+0000 7f48c1f0b640 10 monclient: shutdown 2024-03-29T10:39:40.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.752+0000 7f48c1f0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:40.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.752+0000 7f48c1f0b640 1 -- 172.21.15.84:0/1625798800 >> v1:172.21.15.84:6790/0 conn(0x7f48b4006800 legacy=0x7f48b40cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:40.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.753+0000 7f48c1f0b640 1 -- 172.21.15.84:0/1625798800 shutdown_connections 2024-03-29T10:39:40.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.753+0000 7f48c1f0b640 1 -- 172.21.15.84:0/1625798800 >> 172.21.15.84:0/1625798800 conn(0x7f48b40058f0 msgr2=0x7f48b4005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:40.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.754+0000 7f48c1f0b640 1 -- 172.21.15.84:0/1625798800 shutdown_connections 2024-03-29T10:39:40.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.754+0000 7f48c1f0b640 1 -- 172.21.15.84:0/1625798800 wait complete. 2024-03-29T10:39:40.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.755+0000 7f48c1f0b640 10 monclient: build_initial_monmap 2024-03-29T10:39:40.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.755+0000 7f48c1f0b640 10 monclient: monmap: 2024-03-29T10:39:40.741 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:39:40.742 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:40.742 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:39:40.742 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:39:40.742 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:40.742 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:40.742 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:39:40.742 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:39:40.742 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:39:40.742 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:40.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.755+0000 7f48c1f0b640 1 librados: starting msgr at 2024-03-29T10:39:40.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.755+0000 7f48c1f0b640 1 librados: starting objecter 2024-03-29T10:39:40.742 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.755+0000 7f48c1f0b640 1 Processor -- start 2024-03-29T10:39:40.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.756+0000 7f48c1f0b640 1 -- start start 2024-03-29T10:39:40.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.756+0000 7f48c1f0b640 1 librados: setting wanted keys 2024-03-29T10:39:40.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.756+0000 7f48c1f0b640 1 librados: calling monclient init 2024-03-29T10:39:40.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.756+0000 7f48c1f0b640 10 monclient: init 2024-03-29T10:39:40.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.757+0000 7f48c1f0b640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:40.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.757+0000 7f48c1f0b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:39:40.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.757+0000 7f48c1f0b640 10 monclient(hunting): picked mon.noname-c con 0x7f48b4006800 addr v1:172.21.15.84:6790/0 2024-03-29T10:39:40.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.758+0000 7f48c1f0b640 10 monclient(hunting): picked mon.noname-a con 0x7f48b4031b60 addr v1:172.21.15.84:6789/0 2024-03-29T10:39:40.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.758+0000 7f48c1f0b640 10 monclient(hunting): picked mon.noname-b con 0x7f48b402d560 addr v1:172.21.15.86:6789/0 2024-03-29T10:39:40.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.758+0000 7f48c1f0b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48b40304d0 con 0x7f48b4031b60 2024-03-29T10:39:40.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.758+0000 7f48c1f0b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48b402e550 con 0x7f48b402d560 2024-03-29T10:39:40.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.758+0000 7f48c1f0b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48b402e830 con 0x7f48b4006800 2024-03-29T10:39:40.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.758+0000 7f48c1f0b640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:40.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.758+0000 7f48c1f0b640 10 monclient(hunting): authenticate will time out at 1083.022422s 2024-03-29T10:39:40.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.759+0000 7f48c0f09640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f48b4006800 0x7f48b40c00f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:36834/0 (socket says 172.21.15.84:36834) 2024-03-29T10:39:40.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.759+0000 7f48c0f09640 1 -- 172.21.15.84:0/3263796935 learned_addr learned my addr 172.21.15.84:0/3263796935 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:39:40.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.762+0000 7f48b97fa640 10 client.?.objecter ms_handle_connect 0x7f48b4006800 2024-03-29T10:39:40.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.762+0000 7f48b97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:40.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.763+0000 7f48b97fa640 1 -- 172.21.15.84:0/3263796935 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109496966 0 0) 0x7f48b402e830 con 0x7f48b4006800 2024-03-29T10:39:40.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.763+0000 7f48b97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:39:40.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.763+0000 7f48b97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:40.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.763+0000 7f48b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:40.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.763+0000 7f48b97fa640 10 monclient(hunting): my global_id is 4436 2024-03-29T10:39:40.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.763+0000 7f48b97fa640 1 -- 172.21.15.84:0/3263796935 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48a0004eb0 con 0x7f48b4006800 2024-03-29T10:39:40.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.764+0000 7f48b97fa640 10 client.?.objecter ms_handle_connect 0x7f48b402d560 2024-03-29T10:39:40.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.764+0000 7f48b97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:40.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.765+0000 7f48b97fa640 1 -- 172.21.15.84:0/3263796935 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107369610 0 0) 0x7f48b402e550 con 0x7f48b402d560 2024-03-29T10:39:40.753 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.765+0000 7f48b97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:39:40.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.765+0000 7f48b97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:40.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.765+0000 7f48b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:40.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.765+0000 7f48b97fa640 10 monclient(hunting): my global_id is 4381 2024-03-29T10:39:40.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.765+0000 7f48b97fa640 1 -- 172.21.15.84:0/3263796935 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48b402e830 con 0x7f48b402d560 2024-03-29T10:39:40.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.765+0000 7f48b97fa640 1 -- 172.21.15.84:0/3263796935 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1902971719 0 0) 0x7f48a0004eb0 con 0x7f48b4006800 2024-03-29T10:39:40.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.765+0000 7f48b97fa640 1 -- 172.21.15.84:0/3263796935 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48b402e550 con 0x7f48b4006800 2024-03-29T10:39:40.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.765+0000 7f48b97fa640 1 -- 172.21.15.84:0/3263796935 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f48a8001be0 con 0x7f48b4006800 2024-03-29T10:39:40.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.765+0000 7f48b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:40.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.765+0000 7f48b97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:40.754 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.765+0000 7f48b97fa640 10 monclient(hunting): dump: 2024-03-29T10:39:40.754 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:40.754 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:40.754 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:40.755 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:40.755 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:40.755 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:40.755 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:40.755 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:40.755 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:40.755 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:40.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.765+0000 7f48b97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:40.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.765+0000 7f48b97fa640 1 -- 172.21.15.84:0/3263796935 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 72886295 0 0) 0x7f48b402e830 con 0x7f48b402d560 2024-03-29T10:39:40.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.765+0000 7f48b97fa640 1 -- 172.21.15.84:0/3263796935 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48a0004eb0 con 0x7f48b402d560 2024-03-29T10:39:40.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.765+0000 7f48b97fa640 1 -- 172.21.15.84:0/3263796935 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f48bc04fdd0 con 0x7f48b402d560 2024-03-29T10:39:40.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.765+0000 7f48b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:40.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.765+0000 7f48b97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:39:40.755 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.765+0000 7f48b97fa640 10 monclient(hunting): dump: 2024-03-29T10:39:40.756 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:40.756 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:40.756 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:40.756 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:40.756 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:40.756 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:40.756 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:40.756 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:40.756 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:40.756 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:40.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.765+0000 7f48b97fa640 1 -- 172.21.15.84:0/3263796935 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1231538666 0 0) 0x7f48b402e550 con 0x7f48b4006800 2024-03-29T10:39:40.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.765+0000 7f48b97fa640 1 -- 172.21.15.84:0/3263796935 >> v1:172.21.15.86:6789/0 conn(0x7f48b402d560 legacy=0x7f48b4164870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:40.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.765+0000 7f48b97fa640 1 -- 172.21.15.84:0/3263796935 >> v1:172.21.15.84:6789/0 conn(0x7f48b4031b60 legacy=0x7f48b4164160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:40.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.765+0000 7f48b97fa640 10 monclient: _finish_hunting 0 2024-03-29T10:39:40.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.765+0000 7f48b97fa640 1 monclient: found mon.c 2024-03-29T10:39:40.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.766+0000 7f48b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:40.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.766+0000 7f48b97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:39:40.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.766+0000 7f48b97fa640 1 -- 172.21.15.84:0/3263796935 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48b402eb10 con 0x7f48b4006800 2024-03-29T10:39:40.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.766+0000 7f48b97fa640 10 monclient: _finish_auth 0 2024-03-29T10:39:40.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.766+0000 7f48b97fa640 10 monclient: _check_auth_tickets 2024-03-29T10:39:40.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.766+0000 7f48b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:40.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.766+0000 7f48c1f0b640 5 monclient: authenticate success, global_id 4436 2024-03-29T10:39:40.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.766+0000 7f48c1f0b640 10 monclient: _renew_subs 2024-03-29T10:39:40.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.766+0000 7f48c1f0b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:39:40.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.766+0000 7f48c1f0b640 1 -- 172.21.15.84:0/3263796935 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48b40c0800 con 0x7f48b4006800 2024-03-29T10:39:40.757 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.766+0000 7f48b97fa640 1 -- 172.21.15.84:0/3263796935 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48a8004c60 con 0x7f48b4006800 2024-03-29T10:39:40.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.766+0000 7f48b97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:40.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.766+0000 7f48c1f0b640 10 client.4436.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:40.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.766+0000 7f48baffd640 4 set_mon_vals no callback set 2024-03-29T10:39:40.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.767+0000 7f48c1f0b640 10 monclient: _renew_subs 2024-03-29T10:39:40.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.767+0000 7f48c1f0b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:39:40.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.767+0000 7f48c1f0b640 1 -- 172.21.15.84:0/3263796935 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f48b40c83f0 con 0x7f48b4006800 2024-03-29T10:39:40.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.767+0000 7f48c1f0b640 1 librados: init done 2024-03-29T10:39:40.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.767+0000 7f48b97fa640 1 -- 172.21.15.84:0/3263796935 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f48a8004e50 con 0x7f48b4006800 2024-03-29T10:39:40.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.767+0000 7f48b97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:40.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.767+0000 7f48b97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:40.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.767+0000 7f48b97fa640 10 monclient: dump: 2024-03-29T10:39:40.758 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:40.758 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:40.759 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:40.759 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:40.759 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:40.759 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:40.759 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:40.759 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:40.759 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:40.759 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:40.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.768+0000 7f48c1f0b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:40.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.768+0000 7f48c1f0b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:40.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.768+0000 7f48c1f0b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:39:40.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.768+0000 7f48c1f0b640 1 -- 172.21.15.84:0/3263796935 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48b4030120 con 0x7f48b4006800 2024-03-29T10:39:40.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.771+0000 7f48b97fa640 1 -- 172.21.15.84:0/3263796935 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f48a801ae20 con 0x7f48b4006800 2024-03-29T10:39:40.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.772+0000 7f48b97fa640 1 -- 172.21.15.84:0/3263796935 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f48a8084ce0 con 0x7f48b4006800 2024-03-29T10:39:40.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.772+0000 7f48b97fa640 10 client.4436.objecter ms_dispatch 0x7f48bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:40.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.772+0000 7f48b97fa640 3 client.4436.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:40.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.772+0000 7f48b97fa640 3 client.4436.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:40.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.772+0000 7f48b97fa640 20 client.4436.objecter dump_active .. 0 homeless 2024-03-29T10:39:40.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.772+0000 7f48bbfff640 10 monclient: get_auth_request con 0x7f48a006dd00 auth_method 0 2024-03-29T10:39:40.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.772+0000 7f48b97fa640 10 client.4436.objecter ms_handle_connect 0x7f48a006dd00 2024-03-29T10:39:40.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.780+0000 7f48b97fa640 1 -- 172.21.15.84:0/3263796935 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f48a80505c0 con 0x7f48b4006800 2024-03-29T10:39:40.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.780+0000 7f48b97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:40.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:40.780+0000 7f48b97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:41.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:41.108+0000 7f48c1f0b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:41.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:41.108+0000 7f48c1f0b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:41.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:41.108+0000 7f48c1f0b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:39:41.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:41.108+0000 7f48c1f0b640 1 -- 172.21.15.84:0/3263796935 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48b4004cd0 con 0x7f48b4006800 2024-03-29T10:39:41.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:41.112+0000 7f48b97fa640 1 -- 172.21.15.84:0/3263796935 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f48a80524f0 con 0x7f48b4006800 2024-03-29T10:39:41.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:41.112+0000 7f48b97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:41.096 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:41.112+0000 7f48b97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:41.096 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:39:41.096 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:41.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:41.116+0000 7f48c1f0b640 10 librados: watch_flush enter 2024-03-29T10:39:41.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:41.116+0000 7f48c1f0b640 10 librados: watch_flush exit 2024-03-29T10:39:41.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:41.116+0000 7f48c1f0b640 20 client.4436.objecter shutdown clearing up homeless session... 2024-03-29T10:39:41.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:41.116+0000 7f48c1f0b640 10 client.4436.objecter successfully canceled tick 2024-03-29T10:39:41.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:41.116+0000 7f48c1f0b640 1 -- 172.21.15.84:0/3263796935 >> v1:172.21.15.86:6816/3337225337 conn(0x7f48a006dd00 legacy=0x7f48a00701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:41.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:41.116+0000 7f48c1f0b640 10 monclient: shutdown 2024-03-29T10:39:41.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:41.116+0000 7f48c1f0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:41.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:41.116+0000 7f48c1f0b640 1 -- 172.21.15.84:0/3263796935 >> v1:172.21.15.84:6790/0 conn(0x7f48b4006800 legacy=0x7f48b40c00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:41.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:41.116+0000 7f48c1f0b640 1 -- 172.21.15.84:0/3263796935 shutdown_connections 2024-03-29T10:39:41.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:41.117+0000 7f48c1f0b640 1 -- 172.21.15.84:0/3263796935 >> 172.21.15.84:0/3263796935 conn(0x7f48b40058f0 msgr2=0x7f48b4005cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:41.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:41.117+0000 7f48c1f0b640 1 -- 172.21.15.84:0/3263796935 shutdown_connections 2024-03-29T10:39:41.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:41.117+0000 7f48c1f0b640 1 -- 172.21.15.84:0/3263796935 wait complete. 2024-03-29T10:39:41.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:41.117+0000 7f48c1f0b640 1 librados: shutdown 2024-03-29T10:39:41.118 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:47.926 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:48.082 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.096+0000 7fac7780f640 10 monclient: get_monmap_and_config 2024-03-29T10:39:48.082 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.096+0000 7fac7780f640 10 monclient: build_initial_monmap 2024-03-29T10:39:48.082 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.097+0000 7fac7780f640 10 monclient: monmap: 2024-03-29T10:39:48.082 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:39:48.082 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:48.082 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:39:48.097964+0000 2024-03-29T10:39:48.082 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:39:48.097964+0000 2024-03-29T10:39:48.082 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:48.082 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:48.082 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:39:48.082 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:39:48.083 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:39:48.083 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:48.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.098+0000 7fac7780f640 1 Processor -- start 2024-03-29T10:39:48.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.098+0000 7fac7780f640 1 -- start start 2024-03-29T10:39:48.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.098+0000 7fac7780f640 10 monclient: init 2024-03-29T10:39:48.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.099+0000 7fac7780f640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:48.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.099+0000 7fac7780f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:39:48.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.099+0000 7fac7780f640 10 monclient(hunting): picked mon.noname-c con 0x7fac700aa2a0 addr v1:172.21.15.86:6789/0 2024-03-29T10:39:48.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.099+0000 7fac7780f640 10 monclient(hunting): picked mon.noname-b con 0x7fac70141f20 addr v1:172.21.15.84:6790/0 2024-03-29T10:39:48.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.100+0000 7fac7780f640 10 monclient(hunting): picked mon.noname-a con 0x7fac7013b3c0 addr v1:172.21.15.84:6789/0 2024-03-29T10:39:48.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.102+0000 7fac7780f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac7013e5c0 con 0x7fac7013b3c0 2024-03-29T10:39:48.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.102+0000 7fac7780f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac700ab190 con 0x7fac700aa2a0 2024-03-29T10:39:48.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.102+0000 7fac7780f640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac700ab470 con 0x7fac70141f20 2024-03-29T10:39:48.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.102+0000 7fac7780f640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:48.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.102+0000 7fac755ab640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fac700aa2a0 0x7fac700ad270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:34349/0 (socket says 172.21.15.84:34349) 2024-03-29T10:39:48.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.102+0000 7fac755ab640 1 -- 172.21.15.84:0/2316980132 learned_addr learned my addr 172.21.15.84:0/2316980132 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:39:48.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.102+0000 7fac7780f640 10 monclient(hunting): authenticate will time out at 1090.366002s 2024-03-29T10:39:48.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.103+0000 7fac67fff640 1 -- 172.21.15.84:0/2316980132 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 14834917 0 0) 0x7fac700ab470 con 0x7fac70141f20 2024-03-29T10:39:48.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.103+0000 7fac67fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:48.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.103+0000 7fac67fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:48.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.103+0000 7fac67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:48.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.103+0000 7fac67fff640 10 monclient(hunting): my global_id is 4439 2024-03-29T10:39:48.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.103+0000 7fac67fff640 1 -- 172.21.15.84:0/2316980132 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac5c003520 con 0x7fac70141f20 2024-03-29T10:39:48.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.103+0000 7fac67fff640 1 -- 172.21.15.84:0/2316980132 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 689424203 0 0) 0x7fac700ab190 con 0x7fac700aa2a0 2024-03-29T10:39:48.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.103+0000 7fac67fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:48.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.103+0000 7fac67fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:48.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.103+0000 7fac67fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:48.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.103+0000 7fac67fff640 10 monclient(hunting): my global_id is 4384 2024-03-29T10:39:48.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.103+0000 7fac67fff640 1 -- 172.21.15.84:0/2316980132 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac700ab470 con 0x7fac700aa2a0 2024-03-29T10:39:48.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.104+0000 7fac67fff640 1 -- 172.21.15.84:0/2316980132 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2108889476 0 0) 0x7fac5c003520 con 0x7fac70141f20 2024-03-29T10:39:48.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.104+0000 7fac67fff640 1 -- 172.21.15.84:0/2316980132 >> v1:172.21.15.86:6789/0 conn(0x7fac700aa2a0 legacy=0x7fac700ad270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:48.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.104+0000 7fac67fff640 1 -- 172.21.15.84:0/2316980132 >> v1:172.21.15.84:6789/0 conn(0x7fac7013b3c0 legacy=0x7fac7013b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:48.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.104+0000 7fac67fff640 10 monclient: _finish_hunting 0 2024-03-29T10:39:48.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.104+0000 7fac67fff640 1 monclient: found mon.noname-b 2024-03-29T10:39:48.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.104+0000 7fac67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:48.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.104+0000 7fac67fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:39:48.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.104+0000 7fac67fff640 1 -- 172.21.15.84:0/2316980132 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac70142a50 con 0x7fac70141f20 2024-03-29T10:39:48.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.104+0000 7fac67fff640 10 monclient: _finish_auth 0 2024-03-29T10:39:48.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.104+0000 7fac67fff640 10 monclient: _check_auth_tickets 2024-03-29T10:39:48.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.104+0000 7fac67fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:48.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.104+0000 7fac67fff640 1 -- 172.21.15.84:0/2316980132 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fac600016b0 con 0x7fac70141f20 2024-03-29T10:39:48.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.104+0000 7fac67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:48.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.104+0000 7fac67fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:48.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.104+0000 7fac67fff640 10 monclient: dump: 2024-03-29T10:39:48.093 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:48.093 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:48.093 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:48.093 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:48.093 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:48.093 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:48.093 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:48.094 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:48.094 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:48.094 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:48.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.104+0000 7fac67fff640 1 -- 172.21.15.84:0/2316980132 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac60001810 con 0x7fac70141f20 2024-03-29T10:39:48.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.104+0000 7fac67fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:48.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.104+0000 7fac67fff640 1 -- 172.21.15.84:0/2316980132 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fac60001c20 con 0x7fac70141f20 2024-03-29T10:39:48.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.104+0000 7fac67fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:48.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.104+0000 7fac67fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:48.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.104+0000 7fac67fff640 10 monclient: dump: 2024-03-29T10:39:48.094 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:48.094 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:48.094 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:48.094 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:48.094 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:48.095 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:48.095 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:48.095 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:48.095 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:48.095 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:48.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.104+0000 7fac7780f640 5 monclient: authenticate success, global_id 4439 2024-03-29T10:39:48.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.104+0000 7fac7780f640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:48.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.104+0000 7fac7780f640 4 set_mon_vals no callback set 2024-03-29T10:39:48.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.105+0000 7fac7780f640 10 monclient: shutdown 2024-03-29T10:39:48.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.105+0000 7fac7780f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:48.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.105+0000 7fac7780f640 1 -- 172.21.15.84:0/2316980132 >> v1:172.21.15.84:6790/0 conn(0x7fac70141f20 legacy=0x7fac70142340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:48.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.106+0000 7fac7780f640 1 -- 172.21.15.84:0/2316980132 shutdown_connections 2024-03-29T10:39:48.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.106+0000 7fac7780f640 1 -- 172.21.15.84:0/2316980132 >> 172.21.15.84:0/2316980132 conn(0x7fac700b35e0 msgr2=0x7fac700b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:48.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.107+0000 7fac7780f640 1 -- 172.21.15.84:0/2316980132 shutdown_connections 2024-03-29T10:39:48.095 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.107+0000 7fac7780f640 1 -- 172.21.15.84:0/2316980132 wait complete. 2024-03-29T10:39:48.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.108+0000 7fac7780f640 10 monclient: build_initial_monmap 2024-03-29T10:39:48.097 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.109+0000 7fac7780f640 10 monclient: monmap: 2024-03-29T10:39:48.097 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:39:48.097 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:48.097 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:39:48.098 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:39:48.098 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:48.098 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:48.098 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:39:48.098 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:39:48.098 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:39:48.098 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:48.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.110+0000 7fac7780f640 1 librados: starting msgr at 2024-03-29T10:39:48.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.111+0000 7fac7780f640 1 librados: starting objecter 2024-03-29T10:39:48.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.111+0000 7fac7780f640 1 Processor -- start 2024-03-29T10:39:48.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.111+0000 7fac7780f640 1 -- start start 2024-03-29T10:39:48.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.111+0000 7fac7780f640 1 librados: setting wanted keys 2024-03-29T10:39:48.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.111+0000 7fac7780f640 1 librados: calling monclient init 2024-03-29T10:39:48.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.111+0000 7fac7780f640 10 monclient: init 2024-03-29T10:39:48.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.112+0000 7fac7780f640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:48.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.112+0000 7fac7780f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:39:48.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.112+0000 7fac7780f640 10 monclient(hunting): picked mon.noname-a con 0x7fac700aa2a0 addr v1:172.21.15.84:6789/0 2024-03-29T10:39:48.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.112+0000 7fac7780f640 10 monclient(hunting): picked mon.noname-c con 0x7fac7013b3c0 addr v1:172.21.15.84:6790/0 2024-03-29T10:39:48.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.112+0000 7fac7780f640 10 monclient(hunting): picked mon.noname-b con 0x7fac70141f20 addr v1:172.21.15.86:6789/0 2024-03-29T10:39:48.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.112+0000 7fac7780f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac701405b0 con 0x7fac700aa2a0 2024-03-29T10:39:48.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.112+0000 7fac7780f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac700b11d0 con 0x7fac70141f20 2024-03-29T10:39:48.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.112+0000 7fac7780f640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac700b1360 con 0x7fac7013b3c0 2024-03-29T10:39:48.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.112+0000 7fac7780f640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:48.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.112+0000 7fac7780f640 10 monclient(hunting): authenticate will time out at 1090.375978s 2024-03-29T10:39:48.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.115+0000 7fac75dac640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fac70141f20 0x7fac701e6e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:34439/0 (socket says 172.21.15.84:34439) 2024-03-29T10:39:48.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.115+0000 7fac75dac640 1 -- 172.21.15.84:0/1295571294 learned_addr learned my addr 172.21.15.84:0/1295571294 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:39:48.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.115+0000 7fac65ffb640 10 client.?.objecter ms_handle_connect 0x7fac7013b3c0 2024-03-29T10:39:48.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.115+0000 7fac65ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:48.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.116+0000 7fac65ffb640 10 client.?.objecter ms_handle_connect 0x7fac70141f20 2024-03-29T10:39:48.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.116+0000 7fac65ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:48.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.116+0000 7fac65ffb640 1 -- 172.21.15.84:0/1295571294 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266557443 0 0) 0x7fac70142a50 con 0x7fac7013b3c0 2024-03-29T10:39:48.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.116+0000 7fac65ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:39:48.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.116+0000 7fac65ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:48.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.116+0000 7fac65ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:48.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.116+0000 7fac65ffb640 10 monclient(hunting): my global_id is 4442 2024-03-29T10:39:48.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.116+0000 7fac65ffb640 1 -- 172.21.15.84:0/1295571294 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac54004eb0 con 0x7fac7013b3c0 2024-03-29T10:39:48.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.116+0000 7fac65ffb640 10 client.?.objecter ms_handle_connect 0x7fac700aa2a0 2024-03-29T10:39:48.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.116+0000 7fac65ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:48.108 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.116+0000 7fac65ffb640 1 -- 172.21.15.84:0/1295571294 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2154383644 0 0) 0x7fac54004eb0 con 0x7fac7013b3c0 2024-03-29T10:39:48.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.116+0000 7fac65ffb640 1 -- 172.21.15.84:0/1295571294 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac54005300 con 0x7fac7013b3c0 2024-03-29T10:39:48.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.116+0000 7fac65ffb640 1 -- 172.21.15.84:0/1295571294 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fac60001980 con 0x7fac7013b3c0 2024-03-29T10:39:48.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.116+0000 7fac65ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:48.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.116+0000 7fac65ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:48.109 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.116+0000 7fac65ffb640 10 monclient(hunting): dump: 2024-03-29T10:39:48.109 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:48.109 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:48.109 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:48.109 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:48.109 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:48.109 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:48.109 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:48.109 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:48.110 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:48.110 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:48.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.116+0000 7fac65ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:48.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.117+0000 7fac65ffb640 1 -- 172.21.15.84:0/1295571294 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1682539930 0 0) 0x7fac54005300 con 0x7fac7013b3c0 2024-03-29T10:39:48.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.117+0000 7fac65ffb640 1 -- 172.21.15.84:0/1295571294 >> v1:172.21.15.86:6789/0 conn(0x7fac70141f20 legacy=0x7fac701e6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:48.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.117+0000 7fac65ffb640 1 -- 172.21.15.84:0/1295571294 >> v1:172.21.15.84:6789/0 conn(0x7fac700aa2a0 legacy=0x7fac70133790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:48.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.117+0000 7fac65ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:39:48.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.117+0000 7fac65ffb640 1 monclient: found mon.c 2024-03-29T10:39:48.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.117+0000 7fac65ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:48.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.117+0000 7fac65ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:39:48.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.117+0000 7fac65ffb640 1 -- 172.21.15.84:0/1295571294 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac70133ea0 con 0x7fac7013b3c0 2024-03-29T10:39:48.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.117+0000 7fac65ffb640 10 monclient: _finish_auth 0 2024-03-29T10:39:48.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.117+0000 7fac65ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:39:48.110 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.117+0000 7fac65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:48.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.117+0000 7fac7780f640 5 monclient: authenticate success, global_id 4442 2024-03-29T10:39:48.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.117+0000 7fac7780f640 10 monclient: _renew_subs 2024-03-29T10:39:48.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.117+0000 7fac7780f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:39:48.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.117+0000 7fac7780f640 1 -- 172.21.15.84:0/1295571294 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac70134090 con 0x7fac7013b3c0 2024-03-29T10:39:48.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.117+0000 7fac65ffb640 1 -- 172.21.15.84:0/1295571294 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac600035f0 con 0x7fac7013b3c0 2024-03-29T10:39:48.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.117+0000 7fac65ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:48.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.117+0000 7fac65ffb640 1 -- 172.21.15.84:0/1295571294 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fac60001980 con 0x7fac7013b3c0 2024-03-29T10:39:48.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.117+0000 7fac65ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:48.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.117+0000 7fac65ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:48.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.117+0000 7fac65ffb640 10 monclient: dump: 2024-03-29T10:39:48.111 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:48.111 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:48.111 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:48.111 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:48.112 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:48.112 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:48.112 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:48.112 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:48.112 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:48.112 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:48.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.117+0000 7fac677fe640 4 set_mon_vals no callback set 2024-03-29T10:39:48.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.117+0000 7fac7780f640 10 client.4442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:48.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.117+0000 7fac7780f640 10 monclient: _renew_subs 2024-03-29T10:39:48.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.117+0000 7fac7780f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:39:48.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.117+0000 7fac7780f640 1 -- 172.21.15.84:0/1295571294 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac701ca3f0 con 0x7fac7013b3c0 2024-03-29T10:39:48.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.117+0000 7fac7780f640 1 librados: init done 2024-03-29T10:39:48.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.120+0000 7fac5b7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:48.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.120+0000 7fac5b7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:48.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.120+0000 7fac5b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:39:48.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.120+0000 7fac5b7fe640 1 -- 172.21.15.84:0/1295571294 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac700b2440 con 0x7fac7013b3c0 2024-03-29T10:39:48.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.122+0000 7fac65ffb640 1 -- 172.21.15.84:0/1295571294 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7fac600196d0 con 0x7fac7013b3c0 2024-03-29T10:39:48.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.123+0000 7fac65ffb640 1 -- 172.21.15.84:0/1295571294 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7fac60052fd0 con 0x7fac7013b3c0 2024-03-29T10:39:48.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.123+0000 7fac65ffb640 10 client.4442.objecter ms_dispatch 0x7fac70000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:48.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.123+0000 7fac65ffb640 3 client.4442.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:48.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.123+0000 7fac65ffb640 3 client.4442.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:48.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.123+0000 7fac65ffb640 20 client.4442.objecter dump_active .. 0 homeless 2024-03-29T10:39:48.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.123+0000 7fac755ab640 10 monclient: get_auth_request con 0x7fac5406e240 auth_method 0 2024-03-29T10:39:48.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.126+0000 7fac65ffb640 10 client.4442.objecter ms_handle_connect 0x7fac5406e240 2024-03-29T10:39:48.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.132+0000 7fac65ffb640 1 -- 172.21.15.84:0/1295571294 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fac6004f270 con 0x7fac7013b3c0 2024-03-29T10:39:48.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.132+0000 7fac65ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:48.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.132+0000 7fac65ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:48.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.451+0000 7fac5b7fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:48.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.451+0000 7fac5b7fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:48.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.451+0000 7fac5b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:39:48.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.451+0000 7fac5b7fe640 1 -- 172.21.15.84:0/1295571294 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac700b1360 con 0x7fac7013b3c0 2024-03-29T10:39:48.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.452+0000 7fac65ffb640 1 -- 172.21.15.84:0/1295571294 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7fac6004f270 con 0x7fac7013b3c0 2024-03-29T10:39:48.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.452+0000 7fac65ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:48.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.452+0000 7fac65ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:48.437 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:39:48.437 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:48.439 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.455+0000 7fac5b7fe640 10 librados: watch_flush enter 2024-03-29T10:39:48.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.457+0000 7fac5b7fe640 10 librados: watch_flush exit 2024-03-29T10:39:48.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.457+0000 7fac5b7fe640 20 client.4442.objecter shutdown clearing up homeless session... 2024-03-29T10:39:48.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.457+0000 7fac5b7fe640 10 client.4442.objecter successfully canceled tick 2024-03-29T10:39:48.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.457+0000 7fac5b7fe640 1 -- 172.21.15.84:0/1295571294 >> v1:172.21.15.86:6816/3337225337 conn(0x7fac5406e240 legacy=0x7fac54070700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:48.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.457+0000 7fac5b7fe640 10 monclient: shutdown 2024-03-29T10:39:48.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.457+0000 7fac5b7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:48.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.457+0000 7fac5b7fe640 1 -- 172.21.15.84:0/1295571294 >> v1:172.21.15.84:6790/0 conn(0x7fac7013b3c0 legacy=0x7fac70140db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:48.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.458+0000 7fac5b7fe640 1 -- 172.21.15.84:0/1295571294 shutdown_connections 2024-03-29T10:39:48.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.458+0000 7fac5b7fe640 1 -- 172.21.15.84:0/1295571294 >> 172.21.15.84:0/1295571294 conn(0x7fac700b35e0 msgr2=0x7fac700a97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:48.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.458+0000 7fac5b7fe640 1 -- 172.21.15.84:0/1295571294 shutdown_connections 2024-03-29T10:39:48.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.459+0000 7fac5b7fe640 1 -- 172.21.15.84:0/1295571294 wait complete. 2024-03-29T10:39:48.443 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:48.459+0000 7fac5b7fe640 1 librados: shutdown 2024-03-29T10:39:48.463 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:39:55.271 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:39:55.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.437+0000 7f47660f1640 10 monclient: get_monmap_and_config 2024-03-29T10:39:55.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.437+0000 7f47660f1640 10 monclient: build_initial_monmap 2024-03-29T10:39:55.422 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.437+0000 7f47660f1640 10 monclient: monmap: 2024-03-29T10:39:55.422 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:39:55.422 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:55.422 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:39:55.438746+0000 2024-03-29T10:39:55.422 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:39:55.438746+0000 2024-03-29T10:39:55.422 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:55.422 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:55.423 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:39:55.423 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:39:55.423 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:39:55.423 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:55.423 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.438+0000 7f47660f1640 1 Processor -- start 2024-03-29T10:39:55.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.438+0000 7f47660f1640 1 -- start start 2024-03-29T10:39:55.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.438+0000 7f47660f1640 10 monclient: init 2024-03-29T10:39:55.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.439+0000 7f47660f1640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:55.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.439+0000 7f47660f1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:39:55.424 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.439+0000 7f47660f1640 10 monclient(hunting): picked mon.noname-a con 0x7f47600b4330 addr v1:172.21.15.84:6789/0 2024-03-29T10:39:55.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.441+0000 7f47660f1640 10 monclient(hunting): picked mon.noname-b con 0x7f47601470b0 addr v1:172.21.15.84:6790/0 2024-03-29T10:39:55.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.441+0000 7f47660f1640 10 monclient(hunting): picked mon.noname-c con 0x7f47601439b0 addr v1:172.21.15.86:6789/0 2024-03-29T10:39:55.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.441+0000 7f47660f1640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f476006f2a0 con 0x7f47600b4330 2024-03-29T10:39:55.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.441+0000 7f47660f1640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4760146b90 con 0x7f47601439b0 2024-03-29T10:39:55.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.441+0000 7f47660f1640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47600b2ca0 con 0x7f47601470b0 2024-03-29T10:39:55.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.441+0000 7f47660f1640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:55.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.441+0000 7f47660f1640 10 monclient(hunting): authenticate will time out at 1097.704510s 2024-03-29T10:39:55.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.441+0000 7f475effd640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f47601470b0 0x7f47600b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:58336/0 (socket says 172.21.15.84:58336) 2024-03-29T10:39:55.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.441+0000 7f475effd640 1 -- 172.21.15.84:0/3447528571 learned_addr learned my addr 172.21.15.84:0/3447528571 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:39:55.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.443+0000 7f475e7fc640 1 -- 172.21.15.84:0/3447528571 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 95469619 0 0) 0x7f47600b2ca0 con 0x7f47601470b0 2024-03-29T10:39:55.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.443+0000 7f475e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:39:55.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.443+0000 7f475e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:55.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.443+0000 7f475e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:55.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.443+0000 7f475e7fc640 10 monclient(hunting): my global_id is 4445 2024-03-29T10:39:55.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.443+0000 7f475e7fc640 1 -- 172.21.15.84:0/3447528571 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4748003520 con 0x7f47601470b0 2024-03-29T10:39:55.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.443+0000 7f475e7fc640 1 -- 172.21.15.84:0/3447528571 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3996755455 0 0) 0x7f4748003520 con 0x7f47601470b0 2024-03-29T10:39:55.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.443+0000 7f475e7fc640 1 -- 172.21.15.84:0/3447528571 >> v1:172.21.15.86:6789/0 conn(0x7f47601439b0 legacy=0x7f4760151d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:39:55.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.444+0000 7f475f7fe640 1 -- 172.21.15.84:0/3447528571 >> v1:172.21.15.84:6789/0 conn(0x7f47600b4330 legacy=0x7f47600b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-29T10:39:55.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.444+0000 7f475f7fe640 1 -- 172.21.15.84:0/3447528571 >> v1:172.21.15.84:6789/0 conn(0x7f47600b4330 legacy=0x7f47600b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-29T10:39:55.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.444+0000 7f475f7fe640 1 --1- 172.21.15.84:0/3447528571 >> v1:172.21.15.84:6789/0 conn(0x7f47600b4330 0x7f47600b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify read banner and identify addresses failed 2024-03-29T10:39:55.431 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.444+0000 7f475e7fc640 1 -- 172.21.15.84:0/3447528571 >> v1:172.21.15.84:6789/0 conn(0x7f47600b4330 legacy=0x7f47600b4710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:39:55.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.444+0000 7f475e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:39:55.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.444+0000 7f475e7fc640 1 monclient: found mon.noname-b 2024-03-29T10:39:55.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.444+0000 7f475e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:55.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.444+0000 7f475e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:39:55.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.444+0000 7f475e7fc640 1 -- 172.21.15.84:0/3447528571 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f476006e0c0 con 0x7f47601470b0 2024-03-29T10:39:55.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.445+0000 7f475e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:39:55.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.445+0000 7f475e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:39:55.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.445+0000 7f475e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:55.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.445+0000 7f475e7fc640 1 -- 172.21.15.84:0/3447528571 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4750001690 con 0x7f47601470b0 2024-03-29T10:39:55.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.445+0000 7f47660f1640 5 monclient: authenticate success, global_id 4445 2024-03-29T10:39:55.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.445+0000 7f47660f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:55.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.445+0000 7f475e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:55.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.445+0000 7f475e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:39:55.432 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.445+0000 7f475e7fc640 10 monclient: dump: 2024-03-29T10:39:55.433 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:55.433 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:55.433 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:55.433 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:55.433 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:55.433 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:55.433 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:55.433 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:55.433 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:55.433 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:55.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.445+0000 7f475e7fc640 1 -- 172.21.15.84:0/3447528571 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4750001920 con 0x7f47601470b0 2024-03-29T10:39:55.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.445+0000 7f47660f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:39:55.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.445+0000 7f475e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:55.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.445+0000 7f475e7fc640 1 -- 172.21.15.84:0/3447528571 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4750001cb0 con 0x7f47601470b0 2024-03-29T10:39:55.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.445+0000 7f47660f1640 10 monclient: get_monmap_and_config success 2024-03-29T10:39:55.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.445+0000 7f47660f1640 4 set_mon_vals no callback set 2024-03-29T10:39:55.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.446+0000 7f475e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:55.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.446+0000 7f475e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:55.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.446+0000 7f475e7fc640 10 monclient: dump: 2024-03-29T10:39:55.434 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:55.434 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:55.434 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:55.434 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:55.434 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:55.434 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:55.434 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:55.434 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:55.434 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:55.435 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:55.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.446+0000 7f47660f1640 10 monclient: shutdown 2024-03-29T10:39:55.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.446+0000 7f47660f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:55.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.446+0000 7f47660f1640 1 -- 172.21.15.84:0/3447528571 >> v1:172.21.15.84:6790/0 conn(0x7f47601470b0 legacy=0x7f47600b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:55.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.447+0000 7f47660f1640 1 -- 172.21.15.84:0/3447528571 shutdown_connections 2024-03-29T10:39:55.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.447+0000 7f47660f1640 1 -- 172.21.15.84:0/3447528571 >> 172.21.15.84:0/3447528571 conn(0x7f4760066a00 msgr2=0x7f4760066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:55.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.447+0000 7f47660f1640 1 -- 172.21.15.84:0/3447528571 shutdown_connections 2024-03-29T10:39:55.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.447+0000 7f47660f1640 1 -- 172.21.15.84:0/3447528571 wait complete. 2024-03-29T10:39:55.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.449+0000 7f47660f1640 10 monclient: build_initial_monmap 2024-03-29T10:39:55.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.449+0000 7f47660f1640 10 monclient: monmap: 2024-03-29T10:39:55.436 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:39:55.436 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:39:55.436 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:39:55.436 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:39:55.437 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:39:55.437 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:55.437 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:39:55.437 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:39:55.437 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:39:55.437 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:55.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.450+0000 7f47660f1640 1 librados: starting msgr at 2024-03-29T10:39:55.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.450+0000 7f47660f1640 1 librados: starting objecter 2024-03-29T10:39:55.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.450+0000 7f47660f1640 1 Processor -- start 2024-03-29T10:39:55.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.451+0000 7f47660f1640 1 -- start start 2024-03-29T10:39:55.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.451+0000 7f47660f1640 1 librados: setting wanted keys 2024-03-29T10:39:55.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.451+0000 7f47660f1640 1 librados: calling monclient init 2024-03-29T10:39:55.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.451+0000 7f47660f1640 10 monclient: init 2024-03-29T10:39:55.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.453+0000 7f47660f1640 10 monclient: _reopen_session rank -1 2024-03-29T10:39:55.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.453+0000 7f47660f1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:39:55.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.453+0000 7f47660f1640 10 monclient(hunting): picked mon.noname-b con 0x7f47600b4330 addr v1:172.21.15.86:6789/0 2024-03-29T10:39:55.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.453+0000 7f47660f1640 10 monclient(hunting): picked mon.noname-c con 0x7f47601439b0 addr v1:172.21.15.84:6790/0 2024-03-29T10:39:55.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.453+0000 7f47660f1640 10 monclient(hunting): picked mon.noname-a con 0x7f47601470b0 addr v1:172.21.15.84:6789/0 2024-03-29T10:39:55.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.453+0000 7f475effd640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f47601439b0 0x7f476013fcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:58886/0 (socket says 172.21.15.84:58886) 2024-03-29T10:39:55.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.453+0000 7f475effd640 1 -- 172.21.15.84:0/390538104 learned_addr learned my addr 172.21.15.84:0/390538104 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:39:55.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.454+0000 7f47660f1640 1 -- 172.21.15.84:0/390538104 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f476011fce0 con 0x7f47601470b0 2024-03-29T10:39:55.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.454+0000 7f47660f1640 1 -- 172.21.15.84:0/390538104 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4760140ae0 con 0x7f47600b4330 2024-03-29T10:39:55.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.454+0000 7f47660f1640 1 -- 172.21.15.84:0/390538104 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47601212e0 con 0x7f47601439b0 2024-03-29T10:39:55.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.454+0000 7f47660f1640 10 monclient(hunting): _renew_subs 2024-03-29T10:39:55.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.454+0000 7f47660f1640 10 monclient(hunting): authenticate will time out at 1097.717239s 2024-03-29T10:39:55.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.454+0000 7f4747fff640 10 client.?.objecter ms_handle_connect 0x7f47601439b0 2024-03-29T10:39:55.440 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.454+0000 7f4747fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:39:55.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.455+0000 7f4747fff640 1 -- 172.21.15.84:0/390538104 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899662563 0 0) 0x7f47601212e0 con 0x7f47601439b0 2024-03-29T10:39:55.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.455+0000 7f4747fff640 10 monclient(hunting): _negotiate 2024-03-29T10:39:55.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.455+0000 7f4747fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:39:55.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.455+0000 7f4747fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:39:55.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.455+0000 7f4747fff640 10 monclient(hunting): my global_id is 4448 2024-03-29T10:39:55.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.455+0000 7f4747fff640 1 -- 172.21.15.84:0/390538104 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f474c004eb0 con 0x7f47601439b0 2024-03-29T10:39:55.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.455+0000 7f4747fff640 1 -- 172.21.15.84:0/390538104 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2239255446 0 0) 0x7f474c004eb0 con 0x7f47601439b0 2024-03-29T10:39:55.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.456+0000 7f4747fff640 1 -- 172.21.15.84:0/390538104 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47601212e0 con 0x7f47601439b0 2024-03-29T10:39:55.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.456+0000 7f4747fff640 1 -- 172.21.15.84:0/390538104 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4750001610 con 0x7f47601439b0 2024-03-29T10:39:55.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.456+0000 7f4747fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:55.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.456+0000 7f4747fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:39:55.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.456+0000 7f4747fff640 10 monclient(hunting): dump: 2024-03-29T10:39:55.441 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:55.441 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:55.442 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:55.442 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:55.442 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:55.442 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:55.442 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:55.442 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:55.442 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:55.442 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:55.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.456+0000 7f4747fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:39:55.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.456+0000 7f4747fff640 1 -- 172.21.15.84:0/390538104 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2081364782 0 0) 0x7f47601212e0 con 0x7f47601439b0 2024-03-29T10:39:55.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.456+0000 7f4747fff640 1 -- 172.21.15.84:0/390538104 >> v1:172.21.15.86:6789/0 conn(0x7f47600b4330 legacy=0x7f47601db260 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:39:55.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.458+0000 7f4747fff640 1 -- 172.21.15.84:0/390538104 >> v1:172.21.15.84:6789/0 conn(0x7f47601470b0 legacy=0x7f47601403d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:39:55.444 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.458+0000 7f4747fff640 10 monclient: _finish_hunting 0 2024-03-29T10:39:55.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.458+0000 7f4747fff640 1 monclient: found mon.c 2024-03-29T10:39:55.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.458+0000 7f4747fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:39:55.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.458+0000 7f4747fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:39:55.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.459+0000 7f4747fff640 1 -- 172.21.15.84:0/390538104 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47601db970 con 0x7f47601439b0 2024-03-29T10:39:55.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.459+0000 7f4747fff640 10 monclient: _finish_auth 0 2024-03-29T10:39:55.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.459+0000 7f4747fff640 10 monclient: _check_auth_tickets 2024-03-29T10:39:55.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.459+0000 7f4747fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:39:55.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.459+0000 7f4747fff640 1 -- 172.21.15.84:0/390538104 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4750004c60 con 0x7f47601439b0 2024-03-29T10:39:55.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.459+0000 7f4747fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:39:55.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.459+0000 7f4747fff640 1 -- 172.21.15.84:0/390538104 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4750005070 con 0x7f47601439b0 2024-03-29T10:39:55.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.459+0000 7f4747fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:39:55.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.459+0000 7f4747fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:39:55.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.459+0000 7f4747fff640 10 monclient: dump: 2024-03-29T10:39:55.445 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:39:55.446 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:39:55.446 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:55.446 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:39:55.446 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:39:55.446 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:39:55.446 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:39:55.446 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:39:55.446 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:39:55.446 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:39:55.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.459+0000 7f475dffb640 4 set_mon_vals no callback set 2024-03-29T10:39:55.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.460+0000 7f47660f1640 5 monclient: authenticate success, global_id 4448 2024-03-29T10:39:55.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.460+0000 7f47660f1640 10 monclient: _renew_subs 2024-03-29T10:39:55.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.460+0000 7f47660f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:39:55.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.460+0000 7f47660f1640 1 -- 172.21.15.84:0/390538104 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f476011f770 con 0x7f47601439b0 2024-03-29T10:39:55.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.460+0000 7f47660f1640 10 client.4448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:39:55.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.460+0000 7f47660f1640 10 monclient: _renew_subs 2024-03-29T10:39:55.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.460+0000 7f47660f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:39:55.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.460+0000 7f47660f1640 1 -- 172.21.15.84:0/390538104 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f476011fa20 con 0x7f47601439b0 2024-03-29T10:39:55.447 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.460+0000 7f47660f1640 1 librados: init done 2024-03-29T10:39:55.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.461+0000 7f47660f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:55.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.461+0000 7f47660f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:55.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.461+0000 7f47660f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:39:55.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.461+0000 7f47660f1640 1 -- 172.21.15.84:0/390538104 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4760141110 con 0x7f47601439b0 2024-03-29T10:39:55.448 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.462+0000 7f4747fff640 1 -- 172.21.15.84:0/390538104 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f47500051d0 con 0x7f47601439b0 2024-03-29T10:39:55.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.462+0000 7f4747fff640 1 -- 172.21.15.84:0/390538104 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f4750084cd0 con 0x7f47601439b0 2024-03-29T10:39:55.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.462+0000 7f4747fff640 10 client.4448.objecter ms_dispatch 0x7f4760000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:39:55.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.462+0000 7f4747fff640 3 client.4448.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:39:55.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.462+0000 7f4747fff640 3 client.4448.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:39:55.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.463+0000 7f4747fff640 20 client.4448.objecter dump_active .. 0 homeless 2024-03-29T10:39:55.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.463+0000 7f475f7fe640 10 monclient: get_auth_request con 0x7f474c06dc00 auth_method 0 2024-03-29T10:39:55.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.463+0000 7f4747fff640 10 client.4448.objecter ms_handle_connect 0x7f474c06dc00 2024-03-29T10:39:55.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.469+0000 7f4747fff640 1 -- 172.21.15.84:0/390538104 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f47500505b0 con 0x7f47601439b0 2024-03-29T10:39:55.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.469+0000 7f4747fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:39:55.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.469+0000 7f4747fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:39:55.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.796+0000 7f47660f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:55.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.796+0000 7f47660f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:55.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.796+0000 7f47660f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:39:55.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.796+0000 7f47660f1640 1 -- 172.21.15.84:0/390538104 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4760075f10 con 0x7f47601439b0 2024-03-29T10:39:55.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.796+0000 7f4747fff640 1 -- 172.21.15.84:0/390538104 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f47500524e0 con 0x7f47601439b0 2024-03-29T10:39:55.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.796+0000 7f4747fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:39:55.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.796+0000 7f4747fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:39:55.781 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:39:55.781 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:39:55.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.800+0000 7f47660f1640 10 librados: watch_flush enter 2024-03-29T10:39:55.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.801+0000 7f47660f1640 10 librados: watch_flush exit 2024-03-29T10:39:55.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.801+0000 7f47660f1640 20 client.4448.objecter shutdown clearing up homeless session... 2024-03-29T10:39:55.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.801+0000 7f47660f1640 10 client.4448.objecter successfully canceled tick 2024-03-29T10:39:55.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.801+0000 7f47660f1640 1 -- 172.21.15.84:0/390538104 >> v1:172.21.15.86:6816/3337225337 conn(0x7f474c06dc00 legacy=0x7f474c0700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:55.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.801+0000 7f47660f1640 10 monclient: shutdown 2024-03-29T10:39:55.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.801+0000 7f47660f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:39:55.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.801+0000 7f47660f1640 1 -- 172.21.15.84:0/390538104 >> v1:172.21.15.84:6790/0 conn(0x7f47601439b0 legacy=0x7f476013fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:39:55.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.802+0000 7f47660f1640 1 -- 172.21.15.84:0/390538104 shutdown_connections 2024-03-29T10:39:55.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.802+0000 7f47660f1640 1 -- 172.21.15.84:0/390538104 >> 172.21.15.84:0/390538104 conn(0x7f4760066a00 msgr2=0x7f4760066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:39:55.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.802+0000 7f47660f1640 1 -- 172.21.15.84:0/390538104 shutdown_connections 2024-03-29T10:39:55.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.802+0000 7f47660f1640 1 -- 172.21.15.84:0/390538104 wait complete. 2024-03-29T10:39:55.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:39:55.803+0000 7f47660f1640 1 librados: shutdown 2024-03-29T10:39:55.803 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:02.609 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:02.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.772+0000 7f86c744a640 10 monclient: get_monmap_and_config 2024-03-29T10:40:02.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.772+0000 7f86c744a640 10 monclient: build_initial_monmap 2024-03-29T10:40:02.756 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.772+0000 7f86c744a640 10 monclient: monmap: 2024-03-29T10:40:02.757 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:40:02.757 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:02.757 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:40:02.773607+0000 2024-03-29T10:40:02.757 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:40:02.773607+0000 2024-03-29T10:40:02.757 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:02.757 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:02.757 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:40:02.757 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:40:02.757 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:40:02.757 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:02.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.773+0000 7f86c744a640 1 Processor -- start 2024-03-29T10:40:02.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.774+0000 7f86c744a640 1 -- start start 2024-03-29T10:40:02.758 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.774+0000 7f86c744a640 10 monclient: init 2024-03-29T10:40:02.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.774+0000 7f86c744a640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:02.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.774+0000 7f86c744a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:40:02.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.774+0000 7f86c744a640 10 monclient(hunting): picked mon.noname-a con 0x7f86c01457f0 addr v1:172.21.15.84:6789/0 2024-03-29T10:40:02.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.774+0000 7f86c744a640 10 monclient(hunting): picked mon.noname-b con 0x7f86c013f650 addr v1:172.21.15.84:6790/0 2024-03-29T10:40:02.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.774+0000 7f86c744a640 10 monclient(hunting): picked mon.noname-c con 0x7f86c00a9d70 addr v1:172.21.15.86:6789/0 2024-03-29T10:40:02.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.774+0000 7f86c744a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c00aaef0 con 0x7f86c01457f0 2024-03-29T10:40:02.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.774+0000 7f86c744a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c0145bd0 con 0x7f86c00a9d70 2024-03-29T10:40:02.759 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.774+0000 7f86c744a640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c0145eb0 con 0x7f86c013f650 2024-03-29T10:40:02.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.774+0000 7f86c744a640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:02.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.774+0000 7f86c744a640 10 monclient(hunting): authenticate will time out at 1105.037978s 2024-03-29T10:40:02.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.776+0000 7f86c49e5640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f86c013f650 0x7f86c00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:32888/0 (socket says 172.21.15.84:32888) 2024-03-29T10:40:02.760 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.776+0000 7f86c49e5640 1 -- 172.21.15.84:0/2266837316 learned_addr learned my addr 172.21.15.84:0/2266837316 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:40:02.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.779+0000 7f86b7fff640 1 -- 172.21.15.84:0/2266837316 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1603259956 0 0) 0x7f86c0145eb0 con 0x7f86c013f650 2024-03-29T10:40:02.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.779+0000 7f86b7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:02.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.779+0000 7f86b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:02.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.779+0000 7f86b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:02.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.779+0000 7f86b7fff640 10 monclient(hunting): my global_id is 4451 2024-03-29T10:40:02.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.779+0000 7f86b7fff640 1 -- 172.21.15.84:0/2266837316 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86ac003520 con 0x7f86c013f650 2024-03-29T10:40:02.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.779+0000 7f86b7fff640 1 -- 172.21.15.84:0/2266837316 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2562057788 0 0) 0x7f86ac003520 con 0x7f86c013f650 2024-03-29T10:40:02.763 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.779+0000 7f86b7fff640 1 -- 172.21.15.84:0/2266837316 >> v1:172.21.15.86:6789/0 conn(0x7f86c00a9d70 legacy=0x7f86c014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:02.764 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.779+0000 7f86b7fff640 1 -- 172.21.15.84:0/2266837316 >> v1:172.21.15.84:6789/0 conn(0x7f86c01457f0 legacy=0x7f86c013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:40:02.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.780+0000 7f86b7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:40:02.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.780+0000 7f86b7fff640 1 monclient: found mon.noname-b 2024-03-29T10:40:02.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.780+0000 7f86b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:02.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.780+0000 7f86b7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:40:02.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.780+0000 7f86b7fff640 1 -- 172.21.15.84:0/2266837316 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c0146340 con 0x7f86c013f650 2024-03-29T10:40:02.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.780+0000 7f86b7fff640 10 monclient: _finish_auth 0 2024-03-29T10:40:02.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.780+0000 7f86b7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:40:02.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.780+0000 7f86b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:02.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.780+0000 7f86b7fff640 1 -- 172.21.15.84:0/2266837316 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f86bc001690 con 0x7f86c013f650 2024-03-29T10:40:02.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.780+0000 7f86b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:02.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.780+0000 7f86b7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:02.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.780+0000 7f86b7fff640 10 monclient: dump: 2024-03-29T10:40:02.768 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:02.768 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:02.768 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:02.768 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:02.768 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:02.768 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:02.768 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:02.768 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:02.768 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:02.768 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:02.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.780+0000 7f86c744a640 5 monclient: authenticate success, global_id 4451 2024-03-29T10:40:02.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.780+0000 7f86c744a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:02.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.780+0000 7f86b7fff640 1 -- 172.21.15.84:0/2266837316 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86bc001860 con 0x7f86c013f650 2024-03-29T10:40:02.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.780+0000 7f86b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:02.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.780+0000 7f86c744a640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:02.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.780+0000 7f86c744a640 4 set_mon_vals no callback set 2024-03-29T10:40:02.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.780+0000 7f86b7fff640 1 -- 172.21.15.84:0/2266837316 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f86bc001c70 con 0x7f86c013f650 2024-03-29T10:40:02.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.780+0000 7f86b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:02.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.780+0000 7f86b7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:02.769 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.780+0000 7f86b7fff640 10 monclient: dump: 2024-03-29T10:40:02.769 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:02.769 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:02.769 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:02.769 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:02.769 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:02.770 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:02.770 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:02.770 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:02.770 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:02.770 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:02.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.781+0000 7f86c744a640 10 monclient: shutdown 2024-03-29T10:40:02.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.781+0000 7f86c744a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:02.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.781+0000 7f86c744a640 1 -- 172.21.15.84:0/2266837316 >> v1:172.21.15.84:6790/0 conn(0x7f86c013f650 legacy=0x7f86c00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:02.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.783+0000 7f86c744a640 1 -- 172.21.15.84:0/2266837316 shutdown_connections 2024-03-29T10:40:02.770 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.783+0000 7f86c744a640 1 -- 172.21.15.84:0/2266837316 >> 172.21.15.84:0/2266837316 conn(0x7f86c0066a00 msgr2=0x7f86c0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:02.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.784+0000 7f86c744a640 1 -- 172.21.15.84:0/2266837316 shutdown_connections 2024-03-29T10:40:02.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.784+0000 7f86c744a640 1 -- 172.21.15.84:0/2266837316 wait complete. 2024-03-29T10:40:02.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.785+0000 7f86c744a640 10 monclient: build_initial_monmap 2024-03-29T10:40:02.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.785+0000 7f86c744a640 10 monclient: monmap: 2024-03-29T10:40:02.772 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:40:02.772 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:02.772 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:40:02.772 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:40:02.772 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:02.772 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:02.772 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:40:02.773 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:40:02.773 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:40:02.773 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:02.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.785+0000 7f86c744a640 1 librados: starting msgr at 2024-03-29T10:40:02.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.785+0000 7f86c744a640 1 librados: starting objecter 2024-03-29T10:40:02.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.785+0000 7f86c744a640 1 Processor -- start 2024-03-29T10:40:02.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.785+0000 7f86c744a640 1 -- start start 2024-03-29T10:40:02.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.785+0000 7f86c744a640 1 librados: setting wanted keys 2024-03-29T10:40:02.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.785+0000 7f86c744a640 1 librados: calling monclient init 2024-03-29T10:40:02.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.785+0000 7f86c744a640 10 monclient: init 2024-03-29T10:40:02.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.786+0000 7f86c744a640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:02.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.786+0000 7f86c744a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:40:02.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.786+0000 7f86c744a640 10 monclient(hunting): picked mon.noname-b con 0x7f86c00a9d70 addr v1:172.21.15.86:6789/0 2024-03-29T10:40:02.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.786+0000 7f86c744a640 10 monclient(hunting): picked mon.noname-c con 0x7f86c013f650 addr v1:172.21.15.84:6790/0 2024-03-29T10:40:02.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.786+0000 7f86c744a640 10 monclient(hunting): picked mon.noname-a con 0x7f86c01457f0 addr v1:172.21.15.84:6789/0 2024-03-29T10:40:02.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.786+0000 7f86c744a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c00720a0 con 0x7f86c01457f0 2024-03-29T10:40:02.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.786+0000 7f86c744a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c006fdc0 con 0x7f86c00a9d70 2024-03-29T10:40:02.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.786+0000 7f86c744a640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c0138820 con 0x7f86c013f650 2024-03-29T10:40:02.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.786+0000 7f86c744a640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:02.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.786+0000 7f86c744a640 10 monclient(hunting): authenticate will time out at 1105.049734s 2024-03-29T10:40:02.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.787+0000 7f86c49e5640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f86c013f650 0x7f86c0138110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:33612/0 (socket says 172.21.15.84:33612) 2024-03-29T10:40:02.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.787+0000 7f86c49e5640 1 -- 172.21.15.84:0/2690986817 learned_addr learned my addr 172.21.15.84:0/2690986817 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:40:02.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.787+0000 7f86b5ffb640 10 client.?.objecter ms_handle_connect 0x7f86c013f650 2024-03-29T10:40:02.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.787+0000 7f86b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:02.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.790+0000 7f86b5ffb640 1 -- 172.21.15.84:0/2690986817 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36534234 0 0) 0x7f86c0138820 con 0x7f86c013f650 2024-03-29T10:40:02.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.790+0000 7f86b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:40:02.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.790+0000 7f86b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:02.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.790+0000 7f86b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:02.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.790+0000 7f86b5ffb640 10 monclient(hunting): my global_id is 4454 2024-03-29T10:40:02.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.790+0000 7f86b5ffb640 1 -- 172.21.15.84:0/2690986817 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86ac003760 con 0x7f86c013f650 2024-03-29T10:40:02.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.791+0000 7f86b5ffb640 1 -- 172.21.15.84:0/2690986817 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3907230076 0 0) 0x7f86ac003760 con 0x7f86c013f650 2024-03-29T10:40:02.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.791+0000 7f86b5ffb640 1 -- 172.21.15.84:0/2690986817 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86c0138820 con 0x7f86c013f650 2024-03-29T10:40:02.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.791+0000 7f86b5ffb640 1 -- 172.21.15.84:0/2690986817 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f86bc001b70 con 0x7f86c013f650 2024-03-29T10:40:02.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.791+0000 7f86b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:02.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.791+0000 7f86b5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:02.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.791+0000 7f86b5ffb640 10 monclient(hunting): dump: 2024-03-29T10:40:02.779 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:02.779 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:02.779 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:02.779 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:02.779 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:02.779 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:02.779 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:02.780 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:02.780 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:02.780 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:02.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.791+0000 7f86b5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:02.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.791+0000 7f86b5ffb640 1 -- 172.21.15.84:0/2690986817 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1191452593 0 0) 0x7f86c0138820 con 0x7f86c013f650 2024-03-29T10:40:02.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.791+0000 7f86b5ffb640 1 -- 172.21.15.84:0/2690986817 >> v1:172.21.15.86:6789/0 conn(0x7f86c00a9d70 legacy=0x7f86c013b660 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:40:02.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.791+0000 7f86b5ffb640 1 -- 172.21.15.84:0/2690986817 >> v1:172.21.15.84:6789/0 conn(0x7f86c01457f0 legacy=0x7f86c01dbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:02.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.791+0000 7f86b5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:40:02.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.791+0000 7f86b5ffb640 1 monclient: found mon.c 2024-03-29T10:40:02.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.791+0000 7f86b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:02.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.791+0000 7f86b5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:40:02.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.791+0000 7f86b5ffb640 1 -- 172.21.15.84:0/2690986817 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c01db730 con 0x7f86c013f650 2024-03-29T10:40:02.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.791+0000 7f86b5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:40:02.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.791+0000 7f86b5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:40:02.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.791+0000 7f86b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:02.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.791+0000 7f86c744a640 5 monclient: authenticate success, global_id 4454 2024-03-29T10:40:02.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.791+0000 7f86c744a640 10 monclient: _renew_subs 2024-03-29T10:40:02.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.791+0000 7f86c744a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:40:02.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.791+0000 7f86c744a640 1 -- 172.21.15.84:0/2690986817 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c01dc600 con 0x7f86c013f650 2024-03-29T10:40:02.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.791+0000 7f86b5ffb640 1 -- 172.21.15.84:0/2690986817 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86bc004bf0 con 0x7f86c013f650 2024-03-29T10:40:02.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.791+0000 7f86b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:02.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.791+0000 7f86b5ffb640 1 -- 172.21.15.84:0/2690986817 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f86bc004fc0 con 0x7f86c013f650 2024-03-29T10:40:02.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.791+0000 7f86b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:02.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.791+0000 7f86b5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:02.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.791+0000 7f86b5ffb640 10 monclient: dump: 2024-03-29T10:40:02.781 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:02.781 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:02.781 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:02.781 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:02.781 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:02.782 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:02.782 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:02.782 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:02.782 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:02.782 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:02.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.791+0000 7f86b6ffd640 4 set_mon_vals no callback set 2024-03-29T10:40:02.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.793+0000 7f86c744a640 10 client.4454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:02.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.793+0000 7f86c744a640 10 monclient: _renew_subs 2024-03-29T10:40:02.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.793+0000 7f86c744a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:40:02.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.793+0000 7f86c744a640 1 -- 172.21.15.84:0/2690986817 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c006de90 con 0x7f86c013f650 2024-03-29T10:40:02.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.793+0000 7f86c744a640 1 librados: init done 2024-03-29T10:40:02.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.794+0000 7f86b5ffb640 1 -- 172.21.15.84:0/2690986817 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f86bc005120 con 0x7f86c013f650 2024-03-29T10:40:02.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.794+0000 7f86c744a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:02.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.795+0000 7f86c744a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:02.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.795+0000 7f86c744a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:40:02.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.795+0000 7f86c744a640 1 -- 172.21.15.84:0/2690986817 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c01dbc10 con 0x7f86c013f650 2024-03-29T10:40:02.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.796+0000 7f86c51e6640 10 monclient: get_auth_request con 0x7f86ac06ddd0 auth_method 0 2024-03-29T10:40:02.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.797+0000 7f86b5ffb640 1 -- 172.21.15.84:0/2690986817 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f86bc054b10 con 0x7f86c013f650 2024-03-29T10:40:02.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.797+0000 7f86b5ffb640 10 client.4454.objecter ms_dispatch 0x7f86c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:02.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.797+0000 7f86b5ffb640 3 client.4454.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:02.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.797+0000 7f86b5ffb640 3 client.4454.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:02.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.797+0000 7f86b5ffb640 20 client.4454.objecter dump_active .. 0 homeless 2024-03-29T10:40:02.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.797+0000 7f86b5ffb640 10 client.4454.objecter ms_handle_connect 0x7f86ac06ddd0 2024-03-29T10:40:02.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.804+0000 7f86b5ffb640 1 -- 172.21.15.84:0/2690986817 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f86c01dbc10 con 0x7f86c013f650 2024-03-29T10:40:02.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.804+0000 7f86b5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:02.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:02.804+0000 7f86b5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:03.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:03.139+0000 7f86c744a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:03.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:03.139+0000 7f86c744a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:03.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:03.139+0000 7f86c744a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:40:03.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:03.139+0000 7f86c744a640 1 -- 172.21.15.84:0/2690986817 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86c01dbc10 con 0x7f86c013f650 2024-03-29T10:40:03.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:03.141+0000 7f86b5ffb640 1 -- 172.21.15.84:0/2690986817 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f86c01dbc10 con 0x7f86c013f650 2024-03-29T10:40:03.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:03.141+0000 7f86b5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:03.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:03.141+0000 7f86b5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:03.126 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:40:03.127 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:03.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:03.146+0000 7f86c744a640 10 librados: watch_flush enter 2024-03-29T10:40:03.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:03.146+0000 7f86c744a640 10 librados: watch_flush exit 2024-03-29T10:40:03.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:03.146+0000 7f86c744a640 20 client.4454.objecter shutdown clearing up homeless session... 2024-03-29T10:40:03.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:03.146+0000 7f86c744a640 10 client.4454.objecter successfully canceled tick 2024-03-29T10:40:03.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:03.147+0000 7f86c744a640 1 -- 172.21.15.84:0/2690986817 >> v1:172.21.15.86:6816/3337225337 conn(0x7f86ac06ddd0 legacy=0x7f86ac070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:03.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:03.147+0000 7f86c744a640 10 monclient: shutdown 2024-03-29T10:40:03.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:03.147+0000 7f86c744a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:03.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:03.147+0000 7f86c744a640 1 -- 172.21.15.84:0/2690986817 >> v1:172.21.15.84:6790/0 conn(0x7f86c013f650 legacy=0x7f86c0138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:03.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:03.147+0000 7f86c744a640 1 -- 172.21.15.84:0/2690986817 shutdown_connections 2024-03-29T10:40:03.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:03.147+0000 7f86c744a640 1 -- 172.21.15.84:0/2690986817 >> 172.21.15.84:0/2690986817 conn(0x7f86c0066a00 msgr2=0x7f86c013cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:03.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:03.148+0000 7f86c744a640 1 -- 172.21.15.84:0/2690986817 shutdown_connections 2024-03-29T10:40:03.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:03.148+0000 7f86c744a640 1 -- 172.21.15.84:0/2690986817 wait complete. 2024-03-29T10:40:03.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:03.148+0000 7f86c744a640 1 librados: shutdown 2024-03-29T10:40:03.148 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:09.955 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:10.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.114+0000 7f071363e640 10 monclient: get_monmap_and_config 2024-03-29T10:40:10.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.114+0000 7f071363e640 10 monclient: build_initial_monmap 2024-03-29T10:40:10.098 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.114+0000 7f071363e640 10 monclient: monmap: 2024-03-29T10:40:10.098 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:40:10.098 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:10.098 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:40:10.115138+0000 2024-03-29T10:40:10.098 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:40:10.115138+0000 2024-03-29T10:40:10.098 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:10.098 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:10.098 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:40:10.098 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:40:10.099 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:40:10.099 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:10.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.115+0000 7f071363e640 1 Processor -- start 2024-03-29T10:40:10.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.115+0000 7f071363e640 1 -- start start 2024-03-29T10:40:10.099 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.115+0000 7f071363e640 10 monclient: init 2024-03-29T10:40:10.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.116+0000 7f071363e640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:10.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.116+0000 7f071363e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:40:10.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.116+0000 7f071363e640 10 monclient(hunting): picked mon.noname-c con 0x7f070c1422b0 addr v1:172.21.15.86:6789/0 2024-03-29T10:40:10.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.116+0000 7f071363e640 10 monclient(hunting): picked mon.noname-b con 0x7f070c142690 addr v1:172.21.15.84:6790/0 2024-03-29T10:40:10.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.116+0000 7f071363e640 10 monclient(hunting): picked mon.noname-a con 0x7f070c0b17d0 addr v1:172.21.15.84:6789/0 2024-03-29T10:40:10.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.116+0000 7f071363e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f070c142c80 con 0x7f070c0b17d0 2024-03-29T10:40:10.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.116+0000 7f071363e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f070c0aae40 con 0x7f070c1422b0 2024-03-29T10:40:10.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.116+0000 7f071363e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f070c13fbd0 con 0x7f070c142690 2024-03-29T10:40:10.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.116+0000 7f071363e640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:10.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.116+0000 7f071363e640 10 monclient(hunting): authenticate will time out at 1112.379550s 2024-03-29T10:40:10.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.116+0000 7f0711bdb640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f070c0b17d0 0x7f070c0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:59514/0 (socket says 172.21.15.84:59514) 2024-03-29T10:40:10.100 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.116+0000 7f0711bdb640 1 -- 172.21.15.84:0/3577894028 learned_addr learned my addr 172.21.15.84:0/3577894028 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:40:10.101 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.117+0000 7f0703fff640 1 -- 172.21.15.84:0/3577894028 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1273570071 0 0) 0x7f070c0aae40 con 0x7f070c1422b0 2024-03-29T10:40:10.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.117+0000 7f0703fff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:10.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.117+0000 7f0703fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:10.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.117+0000 7f0703fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:10.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.117+0000 7f0703fff640 10 monclient(hunting): my global_id is 4390 2024-03-29T10:40:10.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.117+0000 7f0703fff640 1 -- 172.21.15.84:0/3577894028 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06f8003520 con 0x7f070c1422b0 2024-03-29T10:40:10.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.117+0000 7f0703fff640 1 -- 172.21.15.84:0/3577894028 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 936835963 0 0) 0x7f070c142c80 con 0x7f070c0b17d0 2024-03-29T10:40:10.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.117+0000 7f0703fff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:10.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.117+0000 7f0703fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:10.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.117+0000 7f0703fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:10.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.117+0000 7f0703fff640 10 monclient(hunting): my global_id is 4419 2024-03-29T10:40:10.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.117+0000 7f0703fff640 1 -- 172.21.15.84:0/3577894028 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f070c0aae40 con 0x7f070c0b17d0 2024-03-29T10:40:10.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.117+0000 7f0703fff640 1 -- 172.21.15.84:0/3577894028 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1458715665 0 0) 0x7f06f8003520 con 0x7f070c1422b0 2024-03-29T10:40:10.102 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.117+0000 7f0703fff640 1 -- 172.21.15.84:0/3577894028 >> v1:172.21.15.84:6790/0 conn(0x7f070c142690 legacy=0x7f070c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:10.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.117+0000 7f0703fff640 1 -- 172.21.15.84:0/3577894028 >> v1:172.21.15.84:6789/0 conn(0x7f070c0b17d0 legacy=0x7f070c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:10.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.117+0000 7f0703fff640 10 monclient: _finish_hunting 0 2024-03-29T10:40:10.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.117+0000 7f0703fff640 1 monclient: found mon.noname-c 2024-03-29T10:40:10.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.117+0000 7f0703fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:10.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.117+0000 7f0703fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-29T10:40:10.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.117+0000 7f0703fff640 1 -- 172.21.15.84:0/3577894028 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f070c151940 con 0x7f070c1422b0 2024-03-29T10:40:10.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.117+0000 7f0703fff640 10 monclient: _finish_auth 0 2024-03-29T10:40:10.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.117+0000 7f0703fff640 10 monclient: _check_auth_tickets 2024-03-29T10:40:10.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.117+0000 7f0703fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:10.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.117+0000 7f0703fff640 1 -- 172.21.15.84:0/3577894028 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0708001690 con 0x7f070c1422b0 2024-03-29T10:40:10.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.117+0000 7f0703fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:10.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.117+0000 7f0703fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:10.103 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.117+0000 7f0703fff640 10 monclient: dump: 2024-03-29T10:40:10.103 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:10.104 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:10.104 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:10.104 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:10.104 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:10.104 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:10.104 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:10.104 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:10.104 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:10.104 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:10.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.117+0000 7f071363e640 5 monclient: authenticate success, global_id 4390 2024-03-29T10:40:10.104 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.117+0000 7f071363e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:10.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.120+0000 7f0703fff640 1 -- 172.21.15.84:0/3577894028 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07080017f0 con 0x7f070c1422b0 2024-03-29T10:40:10.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.120+0000 7f0703fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:10.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.120+0000 7f0703fff640 1 -- 172.21.15.84:0/3577894028 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0708001b80 con 0x7f070c1422b0 2024-03-29T10:40:10.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.120+0000 7f071363e640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:10.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.120+0000 7f0703fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:10.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.120+0000 7f0703fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:10.105 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.120+0000 7f0703fff640 10 monclient: dump: 2024-03-29T10:40:10.105 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:10.105 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:10.105 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:10.105 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:10.105 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:10.105 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:10.106 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:10.106 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:10.106 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:10.106 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:10.106 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.120+0000 7f071363e640 4 set_mon_vals no callback set 2024-03-29T10:40:10.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.121+0000 7f071363e640 10 monclient: shutdown 2024-03-29T10:40:10.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.121+0000 7f071363e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:10.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.121+0000 7f071363e640 1 -- 172.21.15.84:0/3577894028 >> v1:172.21.15.86:6789/0 conn(0x7f070c1422b0 legacy=0x7f070c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:10.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.122+0000 7f071363e640 1 -- 172.21.15.84:0/3577894028 shutdown_connections 2024-03-29T10:40:10.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.122+0000 7f071363e640 1 -- 172.21.15.84:0/3577894028 >> 172.21.15.84:0/3577894028 conn(0x7f070c066a00 msgr2=0x7f070c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:10.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.122+0000 7f071363e640 1 -- 172.21.15.84:0/3577894028 shutdown_connections 2024-03-29T10:40:10.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.122+0000 7f071363e640 1 -- 172.21.15.84:0/3577894028 wait complete. 2024-03-29T10:40:10.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.123+0000 7f071363e640 10 monclient: build_initial_monmap 2024-03-29T10:40:10.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.123+0000 7f071363e640 10 monclient: monmap: 2024-03-29T10:40:10.113 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:40:10.113 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:10.113 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:40:10.113 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:40:10.113 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:10.113 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:10.113 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:40:10.113 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:40:10.113 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:40:10.114 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:10.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.123+0000 7f071363e640 1 librados: starting msgr at 2024-03-29T10:40:10.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.123+0000 7f071363e640 1 librados: starting objecter 2024-03-29T10:40:10.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.123+0000 7f071363e640 1 Processor -- start 2024-03-29T10:40:10.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.123+0000 7f071363e640 1 -- start start 2024-03-29T10:40:10.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.123+0000 7f071363e640 1 librados: setting wanted keys 2024-03-29T10:40:10.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.123+0000 7f071363e640 1 librados: calling monclient init 2024-03-29T10:40:10.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.123+0000 7f071363e640 10 monclient: init 2024-03-29T10:40:10.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.124+0000 7f071363e640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:10.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.124+0000 7f071363e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:40:10.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.124+0000 7f071363e640 10 monclient(hunting): picked mon.noname-a con 0x7f070c0b17d0 addr v1:172.21.15.84:6789/0 2024-03-29T10:40:10.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.124+0000 7f071363e640 10 monclient(hunting): picked mon.noname-c con 0x7f070c1422b0 addr v1:172.21.15.84:6790/0 2024-03-29T10:40:10.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.124+0000 7f07113da640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f070c0b17d0 0x7f070c1310a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:59926/0 (socket says 172.21.15.84:59926) 2024-03-29T10:40:10.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.124+0000 7f07113da640 1 -- 172.21.15.84:0/1159231586 learned_addr learned my addr 172.21.15.84:0/1159231586 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:40:10.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.124+0000 7f071363e640 10 monclient(hunting): picked mon.noname-b con 0x7f070c142690 addr v1:172.21.15.86:6789/0 2024-03-29T10:40:10.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.124+0000 7f071363e640 1 -- 172.21.15.84:0/1159231586 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f070c12f680 con 0x7f070c0b17d0 2024-03-29T10:40:10.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.124+0000 7f071363e640 1 -- 172.21.15.84:0/1159231586 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f070c131b50 con 0x7f070c142690 2024-03-29T10:40:10.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.124+0000 7f0701ffb640 10 client.?.objecter ms_handle_connect 0x7f070c1422b0 2024-03-29T10:40:10.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.124+0000 7f0701ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:10.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.124+0000 7f071363e640 1 -- 172.21.15.84:0/1159231586 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f070c131fa0 con 0x7f070c1422b0 2024-03-29T10:40:10.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.124+0000 7f071363e640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:10.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.124+0000 7f071363e640 10 monclient(hunting): authenticate will time out at 1112.388084s 2024-03-29T10:40:10.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.125+0000 7f0701ffb640 10 client.?.objecter ms_handle_connect 0x7f070c142690 2024-03-29T10:40:10.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.125+0000 7f0701ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:10.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.125+0000 7f0701ffb640 10 client.?.objecter ms_handle_connect 0x7f070c0b17d0 2024-03-29T10:40:10.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.125+0000 7f0701ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:10.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.125+0000 7f0701ffb640 1 -- 172.21.15.84:0/1159231586 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2140804477 0 0) 0x7f070c131fa0 con 0x7f070c1422b0 2024-03-29T10:40:10.115 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.125+0000 7f0701ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:40:10.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.125+0000 7f0701ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:10.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.125+0000 7f0701ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:10.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.125+0000 7f0701ffb640 10 monclient(hunting): my global_id is 4460 2024-03-29T10:40:10.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.125+0000 7f0701ffb640 1 -- 172.21.15.84:0/1159231586 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06f0004eb0 con 0x7f070c1422b0 2024-03-29T10:40:10.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.125+0000 7f0701ffb640 1 -- 172.21.15.84:0/1159231586 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3797731170 0 0) 0x7f070c131b50 con 0x7f070c142690 2024-03-29T10:40:10.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.125+0000 7f0701ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:40:10.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.125+0000 7f0701ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:10.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.125+0000 7f0701ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:10.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.125+0000 7f0701ffb640 10 monclient(hunting): my global_id is 4393 2024-03-29T10:40:10.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.125+0000 7f0701ffb640 1 -- 172.21.15.84:0/1159231586 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f070c131fa0 con 0x7f070c142690 2024-03-29T10:40:10.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.125+0000 7f0701ffb640 1 -- 172.21.15.84:0/1159231586 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 829257317 0 0) 0x7f070c12f680 con 0x7f070c0b17d0 2024-03-29T10:40:10.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.125+0000 7f0701ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:40:10.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.125+0000 7f0701ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:10.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.125+0000 7f0701ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:10.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.125+0000 7f0701ffb640 10 monclient(hunting): my global_id is 4422 2024-03-29T10:40:10.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.125+0000 7f0701ffb640 1 -- 172.21.15.84:0/1159231586 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f070c131b50 con 0x7f070c0b17d0 2024-03-29T10:40:10.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 1 -- 172.21.15.84:0/1159231586 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3470863788 0 0) 0x7f070c131fa0 con 0x7f070c142690 2024-03-29T10:40:10.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 1 -- 172.21.15.84:0/1159231586 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f070c12f680 con 0x7f070c142690 2024-03-29T10:40:10.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 1 -- 172.21.15.84:0/1159231586 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0704001c70 con 0x7f070c142690 2024-03-29T10:40:10.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:10.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:10.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 10 monclient(hunting): dump: 2024-03-29T10:40:10.117 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:10.117 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:10.117 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:10.117 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:10.117 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:10.117 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:10.117 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:10.118 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:10.118 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:10.118 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:10.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:10.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 1 -- 172.21.15.84:0/1159231586 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1863230035 0 0) 0x7f06f0004eb0 con 0x7f070c1422b0 2024-03-29T10:40:10.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 1 -- 172.21.15.84:0/1159231586 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f070c131fa0 con 0x7f070c1422b0 2024-03-29T10:40:10.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 1 -- 172.21.15.84:0/1159231586 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f06fc001890 con 0x7f070c1422b0 2024-03-29T10:40:10.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:10.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:10.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 10 monclient(hunting): dump: 2024-03-29T10:40:10.118 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:10.118 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:10.118 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:10.118 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:10.119 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:10.119 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:10.119 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:10.119 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:10.119 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:10.119 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:10.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 1 -- 172.21.15.84:0/1159231586 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2356626556 0 0) 0x7f070c131b50 con 0x7f070c0b17d0 2024-03-29T10:40:10.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 1 -- 172.21.15.84:0/1159231586 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06f0004eb0 con 0x7f070c0b17d0 2024-03-29T10:40:10.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 1 -- 172.21.15.84:0/1159231586 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0708001b00 con 0x7f070c0b17d0 2024-03-29T10:40:10.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:10.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:10.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 10 monclient(hunting): dump: 2024-03-29T10:40:10.119 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:10.119 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:10.120 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:10.120 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:10.120 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:10.120 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:10.120 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:10.120 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:10.120 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:10.120 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:10.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 1 -- 172.21.15.84:0/1159231586 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2167587505 0 0) 0x7f070c12f680 con 0x7f070c142690 2024-03-29T10:40:10.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 1 -- 172.21.15.84:0/1159231586 >> v1:172.21.15.84:6790/0 conn(0x7f070c1422b0 legacy=0x7f070c12e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:10.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 1 -- 172.21.15.84:0/1159231586 >> v1:172.21.15.84:6789/0 conn(0x7f070c0b17d0 legacy=0x7f070c1310a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:10.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:40:10.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 1 monclient: found mon.b 2024-03-29T10:40:10.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:10.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:40:10.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 1 -- 172.21.15.84:0/1159231586 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f070c132130 con 0x7f070c142690 2024-03-29T10:40:10.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 10 monclient: _finish_auth 0 2024-03-29T10:40:10.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:40:10.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f0701ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:10.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f071363e640 5 monclient: authenticate success, global_id 4393 2024-03-29T10:40:10.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f071363e640 10 monclient: _renew_subs 2024-03-29T10:40:10.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f071363e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:40:10.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.126+0000 7f071363e640 1 -- 172.21.15.84:0/1159231586 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f070c071f40 con 0x7f070c142690 2024-03-29T10:40:10.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.127+0000 7f071363e640 10 client.4393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:10.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.127+0000 7f071363e640 10 monclient: _renew_subs 2024-03-29T10:40:10.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.127+0000 7f071363e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:40:10.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.127+0000 7f071363e640 1 -- 172.21.15.84:0/1159231586 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f070c06e070 con 0x7f070c142690 2024-03-29T10:40:10.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.127+0000 7f071363e640 1 librados: init done 2024-03-29T10:40:10.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.128+0000 7f0701ffb640 1 -- 172.21.15.84:0/1159231586 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f07040020e0 con 0x7f070c142690 2024-03-29T10:40:10.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.128+0000 7f0701ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:10.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.128+0000 7f0701ffb640 1 -- 172.21.15.84:0/1159231586 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0704004d00 con 0x7f070c142690 2024-03-29T10:40:10.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.128+0000 7f0701ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:10.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.128+0000 7f0701ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:10.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.128+0000 7f0701ffb640 10 monclient: dump: 2024-03-29T10:40:10.122 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:10.122 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:10.122 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:10.122 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:10.122 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:10.122 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:10.122 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:10.122 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:10.122 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:10.123 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:10.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.128+0000 7f07037fe640 4 set_mon_vals no callback set 2024-03-29T10:40:10.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.128+0000 7f071363e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:10.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.129+0000 7f071363e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:10.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.129+0000 7f071363e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:40:10.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.129+0000 7f071363e640 1 -- 172.21.15.84:0/1159231586 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06ec005080 con 0x7f070c142690 2024-03-29T10:40:10.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.130+0000 7f0701ffb640 1 -- 172.21.15.84:0/1159231586 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f070401acd0 con 0x7f070c142690 2024-03-29T10:40:10.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.131+0000 7f0701ffb640 1 -- 172.21.15.84:0/1159231586 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f0704084ca0 con 0x7f070c142690 2024-03-29T10:40:10.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.131+0000 7f0701ffb640 10 client.4393.objecter ms_dispatch 0x7f070c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:10.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.131+0000 7f0701ffb640 3 client.4393.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:10.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.131+0000 7f0701ffb640 3 client.4393.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:10.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.131+0000 7f0701ffb640 20 client.4393.objecter dump_active .. 0 homeless 2024-03-29T10:40:10.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.131+0000 7f07113da640 10 monclient: get_auth_request con 0x7f06f006e070 auth_method 0 2024-03-29T10:40:10.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.131+0000 7f0701ffb640 10 client.4393.objecter ms_handle_connect 0x7f06f006e070 2024-03-29T10:40:10.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.140+0000 7f0701ffb640 1 -- 172.21.15.84:0/1159231586 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f07040503b0 con 0x7f070c142690 2024-03-29T10:40:10.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.140+0000 7f0701ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:10.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.140+0000 7f0701ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:10.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.501+0000 7f071363e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:10.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.501+0000 7f071363e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:10.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.501+0000 7f071363e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:40:10.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.501+0000 7f071363e640 1 -- 172.21.15.84:0/1159231586 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06ec005330 con 0x7f070c142690 2024-03-29T10:40:10.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.502+0000 7f0701ffb640 1 -- 172.21.15.84:0/1159231586 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f07040524b0 con 0x7f070c142690 2024-03-29T10:40:10.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.502+0000 7f0701ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:10.485 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.502+0000 7f0701ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:10.486 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:40:10.486 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:10.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.505+0000 7f071363e640 10 librados: watch_flush enter 2024-03-29T10:40:10.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.505+0000 7f071363e640 10 librados: watch_flush exit 2024-03-29T10:40:10.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.505+0000 7f071363e640 20 client.4393.objecter shutdown clearing up homeless session... 2024-03-29T10:40:10.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.505+0000 7f071363e640 10 client.4393.objecter successfully canceled tick 2024-03-29T10:40:10.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.505+0000 7f071363e640 1 -- 172.21.15.84:0/1159231586 >> v1:172.21.15.86:6816/3337225337 conn(0x7f06f006e070 legacy=0x7f06f0070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:10.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.505+0000 7f071363e640 10 monclient: shutdown 2024-03-29T10:40:10.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.505+0000 7f071363e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:10.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.505+0000 7f071363e640 1 -- 172.21.15.84:0/1159231586 >> v1:172.21.15.86:6789/0 conn(0x7f070c142690 legacy=0x7f070c12ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:10.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.506+0000 7f071363e640 1 -- 172.21.15.84:0/1159231586 shutdown_connections 2024-03-29T10:40:10.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.506+0000 7f071363e640 1 -- 172.21.15.84:0/1159231586 >> 172.21.15.84:0/1159231586 conn(0x7f070c066a00 msgr2=0x7f070c13f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:10.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.506+0000 7f071363e640 1 -- 172.21.15.84:0/1159231586 shutdown_connections 2024-03-29T10:40:10.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.506+0000 7f071363e640 1 -- 172.21.15.84:0/1159231586 wait complete. 2024-03-29T10:40:10.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:10.506+0000 7f071363e640 1 librados: shutdown 2024-03-29T10:40:10.505 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:17.312 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:17.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.484+0000 7f4ede520640 10 monclient: get_monmap_and_config 2024-03-29T10:40:17.468 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.484+0000 7f4ede520640 10 monclient: build_initial_monmap 2024-03-29T10:40:17.469 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.484+0000 7f4ede520640 10 monclient: monmap: 2024-03-29T10:40:17.469 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:40:17.469 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:17.469 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:40:17.485632+0000 2024-03-29T10:40:17.469 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:40:17.485632+0000 2024-03-29T10:40:17.469 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:17.469 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:17.469 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:40:17.469 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:40:17.469 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:40:17.469 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:17.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.487+0000 7f4ede520640 1 Processor -- start 2024-03-29T10:40:17.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.487+0000 7f4ede520640 1 -- start start 2024-03-29T10:40:17.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.487+0000 7f4ede520640 10 monclient: init 2024-03-29T10:40:17.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.488+0000 7f4ede520640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:17.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.488+0000 7f4ede520640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:40:17.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.488+0000 7f4ede520640 10 monclient(hunting): picked mon.noname-b con 0x7f4ed81457f0 addr v1:172.21.15.84:6790/0 2024-03-29T10:40:17.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.488+0000 7f4ede520640 10 monclient(hunting): picked mon.noname-a con 0x7f4ed813f650 addr v1:172.21.15.84:6789/0 2024-03-29T10:40:17.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.488+0000 7f4ede520640 10 monclient(hunting): picked mon.noname-c con 0x7f4ed80a9d70 addr v1:172.21.15.86:6789/0 2024-03-29T10:40:17.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.488+0000 7f4ede520640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed80aaef0 con 0x7f4ed813f650 2024-03-29T10:40:17.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.488+0000 7f4ede520640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed8145bd0 con 0x7f4ed80a9d70 2024-03-29T10:40:17.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.488+0000 7f4ede520640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed8145eb0 con 0x7f4ed81457f0 2024-03-29T10:40:17.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.488+0000 7f4ede520640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:17.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.488+0000 7f4ede520640 10 monclient(hunting): authenticate will time out at 1119.751445s 2024-03-29T10:40:17.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.492+0000 7f4ed77fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f4ed813f650 0x7f4ed80b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:42025/0 (socket says 172.21.15.84:42025) 2024-03-29T10:40:17.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.492+0000 7f4ed77fe640 1 -- 172.21.15.84:0/751426261 learned_addr learned my addr 172.21.15.84:0/751426261 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:40:17.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.492+0000 7f4ed6ffd640 1 -- 172.21.15.84:0/751426261 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2108475080 0 0) 0x7f4ed80aaef0 con 0x7f4ed813f650 2024-03-29T10:40:17.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.492+0000 7f4ed6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:17.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.492+0000 7f4ed6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:17.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.492+0000 7f4ed6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:17.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.492+0000 7f4ed6ffd640 10 monclient(hunting): my global_id is 4425 2024-03-29T10:40:17.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.492+0000 7f4ed6ffd640 1 -- 172.21.15.84:0/751426261 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ec0003520 con 0x7f4ed813f650 2024-03-29T10:40:17.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.492+0000 7f4ed6ffd640 1 -- 172.21.15.84:0/751426261 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3917009334 0 0) 0x7f4ec0003520 con 0x7f4ed813f650 2024-03-29T10:40:17.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.492+0000 7f4ed6ffd640 1 -- 172.21.15.84:0/751426261 >> v1:172.21.15.84:6790/0 conn(0x7f4ed81457f0 legacy=0x7f4ed813ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:40:17.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.495+0000 7f4ed6ffd640 1 -- 172.21.15.84:0/751426261 >> v1:172.21.15.86:6789/0 conn(0x7f4ed80a9d70 legacy=0x7f4ed814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:17.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.495+0000 7f4ed6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:40:17.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.495+0000 7f4ed6ffd640 1 monclient: found mon.noname-a 2024-03-29T10:40:17.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.495+0000 7f4ed6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:17.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.495+0000 7f4ed6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T10:40:17.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.495+0000 7f4ed6ffd640 1 -- 172.21.15.84:0/751426261 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed8146340 con 0x7f4ed813f650 2024-03-29T10:40:17.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.495+0000 7f4ed6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:40:17.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.495+0000 7f4ed6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:40:17.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.495+0000 7f4ed6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:17.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.495+0000 7f4ede520640 5 monclient: authenticate success, global_id 4425 2024-03-29T10:40:17.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.495+0000 7f4ede520640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:17.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.496+0000 7f4ed6ffd640 1 -- 172.21.15.84:0/751426261 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4ed0001a10 con 0x7f4ed813f650 2024-03-29T10:40:17.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.496+0000 7f4ed6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:17.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.496+0000 7f4ed6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:40:17.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.496+0000 7f4ed6ffd640 10 monclient: dump: 2024-03-29T10:40:17.481 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:17.481 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:17.482 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:17.482 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:17.482 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:17.482 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:17.482 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:17.482 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:17.482 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:17.482 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:17.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.496+0000 7f4ede520640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:17.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.496+0000 7f4ed6ffd640 1 -- 172.21.15.84:0/751426261 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ed0001ca0 con 0x7f4ed813f650 2024-03-29T10:40:17.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.496+0000 7f4ed6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:17.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.496+0000 7f4ede520640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:17.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.496+0000 7f4ede520640 4 set_mon_vals no callback set 2024-03-29T10:40:17.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.497+0000 7f4ede520640 10 monclient: shutdown 2024-03-29T10:40:17.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.497+0000 7f4ede520640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:17.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.497+0000 7f4ede520640 1 -- 172.21.15.84:0/751426261 >> v1:172.21.15.84:6789/0 conn(0x7f4ed813f650 legacy=0x7f4ed80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:17.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.499+0000 7f4ede520640 1 -- 172.21.15.84:0/751426261 shutdown_connections 2024-03-29T10:40:17.484 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.499+0000 7f4ede520640 1 -- 172.21.15.84:0/751426261 >> 172.21.15.84:0/751426261 conn(0x7f4ed8066a00 msgr2=0x7f4ed8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:17.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.501+0000 7f4ede520640 1 -- 172.21.15.84:0/751426261 shutdown_connections 2024-03-29T10:40:17.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.501+0000 7f4ede520640 1 -- 172.21.15.84:0/751426261 wait complete. 2024-03-29T10:40:17.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.502+0000 7f4ede520640 10 monclient: build_initial_monmap 2024-03-29T10:40:17.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.502+0000 7f4ede520640 10 monclient: monmap: 2024-03-29T10:40:17.489 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:40:17.489 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:17.489 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:40:17.489 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:40:17.489 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:17.489 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:17.489 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:40:17.489 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:40:17.489 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:40:17.489 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:17.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.502+0000 7f4ede520640 1 librados: starting msgr at 2024-03-29T10:40:17.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.502+0000 7f4ede520640 1 librados: starting objecter 2024-03-29T10:40:17.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.502+0000 7f4ede520640 1 Processor -- start 2024-03-29T10:40:17.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.503+0000 7f4ede520640 1 -- start start 2024-03-29T10:40:17.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.503+0000 7f4ede520640 1 librados: setting wanted keys 2024-03-29T10:40:17.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.503+0000 7f4ede520640 1 librados: calling monclient init 2024-03-29T10:40:17.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.503+0000 7f4ede520640 10 monclient: init 2024-03-29T10:40:17.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.503+0000 7f4ede520640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:17.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.503+0000 7f4ede520640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:40:17.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.503+0000 7f4ede520640 10 monclient(hunting): picked mon.noname-b con 0x7f4ed80a9d70 addr v1:172.21.15.86:6789/0 2024-03-29T10:40:17.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.503+0000 7f4ede520640 10 monclient(hunting): picked mon.noname-c con 0x7f4ed813f650 addr v1:172.21.15.84:6790/0 2024-03-29T10:40:17.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.503+0000 7f4ede520640 10 monclient(hunting): picked mon.noname-a con 0x7f4ed81457f0 addr v1:172.21.15.84:6789/0 2024-03-29T10:40:17.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.503+0000 7f4ede520640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed80b5f20 con 0x7f4ed81457f0 2024-03-29T10:40:17.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.503+0000 7f4ede520640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed80b75f0 con 0x7f4ed80a9d70 2024-03-29T10:40:17.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.503+0000 7f4ede520640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed80b7130 con 0x7f4ed813f650 2024-03-29T10:40:17.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.503+0000 7f4ede520640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:17.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.503+0000 7f4ede520640 10 monclient(hunting): authenticate will time out at 1119.766934s 2024-03-29T10:40:17.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.507+0000 7f4ed7fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f4ed80a9d70 0x7f4ed80b5810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:52673/0 (socket says 172.21.15.84:52673) 2024-03-29T10:40:17.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.507+0000 7f4ed7fff640 1 -- 172.21.15.84:0/1855969748 learned_addr learned my addr 172.21.15.84:0/1855969748 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:40:17.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.507+0000 7f4ed4ff9640 10 client.?.objecter ms_handle_connect 0x7f4ed80a9d70 2024-03-29T10:40:17.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.507+0000 7f4ed4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:17.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.507+0000 7f4ed4ff9640 1 -- 172.21.15.84:0/1855969748 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1406322231 0 0) 0x7f4ed80b75f0 con 0x7f4ed80a9d70 2024-03-29T10:40:17.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.507+0000 7f4ed4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:40:17.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.507+0000 7f4ed4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:17.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.507+0000 7f4ed4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:17.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.507+0000 7f4ed4ff9640 10 monclient(hunting): my global_id is 4399 2024-03-29T10:40:17.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.507+0000 7f4ed4ff9640 1 -- 172.21.15.84:0/1855969748 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ec4004eb0 con 0x7f4ed80a9d70 2024-03-29T10:40:17.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.509+0000 7f4ed4ff9640 1 -- 172.21.15.84:0/1855969748 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 29681129 0 0) 0x7f4ec4004eb0 con 0x7f4ed80a9d70 2024-03-29T10:40:17.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.509+0000 7f4ed4ff9640 1 -- 172.21.15.84:0/1855969748 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ec4005300 con 0x7f4ed80a9d70 2024-03-29T10:40:17.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.509+0000 7f4ed4ff9640 1 -- 172.21.15.84:0/1855969748 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4ecc001790 con 0x7f4ed80a9d70 2024-03-29T10:40:17.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.509+0000 7f4ed4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:17.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.509+0000 7f4ed4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:17.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.509+0000 7f4ed4ff9640 10 monclient(hunting): dump: 2024-03-29T10:40:17.494 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:17.494 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:17.494 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:17.494 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:17.494 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:17.494 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:17.494 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:17.494 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:17.495 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:17.495 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:17.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.509+0000 7f4ed4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:17.497 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.511+0000 7f4ed4ff9640 10 client.?.objecter ms_handle_connect 0x7f4ed813f650 2024-03-29T10:40:17.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.511+0000 7f4ed4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:17.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.511+0000 7f4ed4ff9640 10 client.?.objecter ms_handle_connect 0x7f4ed81457f0 2024-03-29T10:40:17.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.511+0000 7f4ed4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:17.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.511+0000 7f4ed4ff9640 1 -- 172.21.15.84:0/1855969748 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2229345241 0 0) 0x7f4ed80b7130 con 0x7f4ed813f650 2024-03-29T10:40:17.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.511+0000 7f4ed4ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:40:17.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.511+0000 7f4ed4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:17.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.511+0000 7f4ed4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:17.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.511+0000 7f4ed4ff9640 10 monclient(hunting): my global_id is 4463 2024-03-29T10:40:17.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.511+0000 7f4ed4ff9640 1 -- 172.21.15.84:0/1855969748 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ec40058d0 con 0x7f4ed813f650 2024-03-29T10:40:17.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.511+0000 7f4ed4ff9640 1 -- 172.21.15.84:0/1855969748 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4124405591 0 0) 0x7f4ec4005300 con 0x7f4ed80a9d70 2024-03-29T10:40:17.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.511+0000 7f4ed4ff9640 1 -- 172.21.15.84:0/1855969748 >> v1:172.21.15.84:6790/0 conn(0x7f4ed813f650 legacy=0x7f4ed80b6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:17.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.511+0000 7f4ed4ff9640 1 -- 172.21.15.84:0/1855969748 >> v1:172.21.15.84:6789/0 conn(0x7f4ed81457f0 legacy=0x7f4ed81ee680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:17.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.511+0000 7f4ed4ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:40:17.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.511+0000 7f4ed4ff9640 1 monclient: found mon.b 2024-03-29T10:40:17.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.511+0000 7f4ed4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:17.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.511+0000 7f4ed4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:40:17.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.511+0000 7f4ed4ff9640 1 -- 172.21.15.84:0/1855969748 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed8141a80 con 0x7f4ed80a9d70 2024-03-29T10:40:17.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.511+0000 7f4ed4ff9640 10 monclient: _finish_auth 0 2024-03-29T10:40:17.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.511+0000 7f4ed4ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:40:17.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.511+0000 7f4ed4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:17.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.512+0000 7f4ed4ff9640 1 -- 172.21.15.84:0/1855969748 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ecc001d60 con 0x7f4ed80a9d70 2024-03-29T10:40:17.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.512+0000 7f4ed4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:17.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.512+0000 7f4ed67fc640 4 set_mon_vals no callback set 2024-03-29T10:40:17.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.512+0000 7f4ed4ff9640 1 -- 172.21.15.84:0/1855969748 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4ecc001790 con 0x7f4ed80a9d70 2024-03-29T10:40:17.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.512+0000 7f4ed4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:17.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.512+0000 7f4ed4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:17.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.512+0000 7f4ed4ff9640 10 monclient: dump: 2024-03-29T10:40:17.499 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:17.500 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:17.500 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:17.500 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:17.500 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:17.500 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:17.500 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:17.500 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:17.500 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:17.500 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:17.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.514+0000 7f4ede520640 5 monclient: authenticate success, global_id 4399 2024-03-29T10:40:17.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.514+0000 7f4ede520640 10 monclient: _renew_subs 2024-03-29T10:40:17.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.514+0000 7f4ede520640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:40:17.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.514+0000 7f4ede520640 1 -- 172.21.15.84:0/1855969748 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed8141cd0 con 0x7f4ed80a9d70 2024-03-29T10:40:17.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.514+0000 7f4ede520640 10 client.4399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:17.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.514+0000 7f4ede520640 10 monclient: _renew_subs 2024-03-29T10:40:17.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.514+0000 7f4ede520640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:40:17.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.514+0000 7f4ede520640 1 -- 172.21.15.84:0/1855969748 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed80b7cf0 con 0x7f4ed80a9d70 2024-03-29T10:40:17.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.514+0000 7f4ede520640 1 librados: init done 2024-03-29T10:40:17.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.516+0000 7f4ede520640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:17.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.516+0000 7f4ede520640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:17.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.516+0000 7f4ede520640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:40:17.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.516+0000 7f4ede520640 1 -- 172.21.15.84:0/1855969748 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed80b7450 con 0x7f4ed80a9d70 2024-03-29T10:40:17.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.517+0000 7f4ed4ff9640 1 -- 172.21.15.84:0/1855969748 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f4ecc01a790 con 0x7f4ed80a9d70 2024-03-29T10:40:17.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.518+0000 7f4ed4ff9640 1 -- 172.21.15.84:0/1855969748 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f4ecc0845c0 con 0x7f4ed80a9d70 2024-03-29T10:40:17.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.518+0000 7f4ed4ff9640 10 client.4399.objecter ms_dispatch 0x7f4ed8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:17.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.518+0000 7f4ed4ff9640 3 client.4399.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:17.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.518+0000 7f4ed4ff9640 3 client.4399.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:17.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.518+0000 7f4ed4ff9640 20 client.4399.objecter dump_active .. 0 homeless 2024-03-29T10:40:17.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.519+0000 7f4ed77fe640 10 monclient: get_auth_request con 0x7f4ec406e420 auth_method 0 2024-03-29T10:40:17.505 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.521+0000 7f4ed4ff9640 10 client.4399.objecter ms_handle_connect 0x7f4ec406e420 2024-03-29T10:40:17.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.525+0000 7f4ed4ff9640 1 -- 172.21.15.84:0/1855969748 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4ed80b7450 con 0x7f4ed80a9d70 2024-03-29T10:40:17.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.525+0000 7f4ed4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:17.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.525+0000 7f4ed4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:17.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.842+0000 7f4ede520640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:17.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.842+0000 7f4ede520640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:17.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.842+0000 7f4ede520640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:40:17.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.842+0000 7f4ede520640 1 -- 172.21.15.84:0/1855969748 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ed8075f10 con 0x7f4ed80a9d70 2024-03-29T10:40:17.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.843+0000 7f4ed4ff9640 1 -- 172.21.15.84:0/1855969748 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f4ed80b7450 con 0x7f4ed80a9d70 2024-03-29T10:40:17.827 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.843+0000 7f4ed4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:17.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.843+0000 7f4ed4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:17.828 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:40:17.828 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:17.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.847+0000 7f4ede520640 10 librados: watch_flush enter 2024-03-29T10:40:17.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.847+0000 7f4ede520640 10 librados: watch_flush exit 2024-03-29T10:40:17.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.847+0000 7f4ede520640 20 client.4399.objecter shutdown clearing up homeless session... 2024-03-29T10:40:17.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.847+0000 7f4ede520640 10 client.4399.objecter successfully canceled tick 2024-03-29T10:40:17.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.848+0000 7f4ede520640 1 -- 172.21.15.84:0/1855969748 >> v1:172.21.15.86:6816/3337225337 conn(0x7f4ec406e420 legacy=0x7f4ec40708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:17.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.848+0000 7f4ede520640 10 monclient: shutdown 2024-03-29T10:40:17.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.848+0000 7f4ede520640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:17.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.848+0000 7f4ede520640 1 -- 172.21.15.84:0/1855969748 >> v1:172.21.15.86:6789/0 conn(0x7f4ed80a9d70 legacy=0x7f4ed80b5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:17.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.850+0000 7f4ede520640 1 -- 172.21.15.84:0/1855969748 shutdown_connections 2024-03-29T10:40:17.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.850+0000 7f4ede520640 1 -- 172.21.15.84:0/1855969748 >> 172.21.15.84:0/1855969748 conn(0x7f4ed8066a00 msgr2=0x7f4ed813b2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:17.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.851+0000 7f4ede520640 1 -- 172.21.15.84:0/1855969748 shutdown_connections 2024-03-29T10:40:17.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.851+0000 7f4ede520640 1 -- 172.21.15.84:0/1855969748 wait complete. 2024-03-29T10:40:17.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:17.851+0000 7f4ede520640 1 librados: shutdown 2024-03-29T10:40:17.853 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:24.660 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:24.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.832+0000 7f98df109640 10 monclient: get_monmap_and_config 2024-03-29T10:40:24.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.832+0000 7f98df109640 10 monclient: build_initial_monmap 2024-03-29T10:40:24.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.832+0000 7f98df109640 10 monclient: monmap: 2024-03-29T10:40:24.816 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:40:24.816 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:24.817 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:40:24.833253+0000 2024-03-29T10:40:24.817 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:40:24.833253+0000 2024-03-29T10:40:24.817 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:24.817 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:24.817 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:40:24.817 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:40:24.817 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:40:24.817 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:24.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.833+0000 7f98df109640 1 Processor -- start 2024-03-29T10:40:24.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.833+0000 7f98df109640 1 -- start start 2024-03-29T10:40:24.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.833+0000 7f98df109640 10 monclient: init 2024-03-29T10:40:24.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.835+0000 7f98df109640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:24.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.835+0000 7f98df109640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:40:24.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.835+0000 7f98df109640 10 monclient(hunting): picked mon.noname-c con 0x7f98d0006a70 addr v1:172.21.15.86:6789/0 2024-03-29T10:40:24.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.835+0000 7f98df109640 10 monclient(hunting): picked mon.noname-a con 0x7f98d002b100 addr v1:172.21.15.84:6789/0 2024-03-29T10:40:24.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.835+0000 7f98df109640 10 monclient(hunting): picked mon.noname-b con 0x7f98d0029c30 addr v1:172.21.15.84:6790/0 2024-03-29T10:40:24.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.835+0000 7f98df109640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d002a270 con 0x7f98d002b100 2024-03-29T10:40:24.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.835+0000 7f98df109640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d002c440 con 0x7f98d0006a70 2024-03-29T10:40:24.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.835+0000 7f98df109640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d00c9440 con 0x7f98d0029c30 2024-03-29T10:40:24.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.835+0000 7f98df109640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:24.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.835+0000 7f98df109640 10 monclient(hunting): authenticate will time out at 1127.098227s 2024-03-29T10:40:24.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.835+0000 7f98de908640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f98d0029c30 0x7f98d00d74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:58988/0 (socket says 172.21.15.84:58988) 2024-03-29T10:40:24.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.835+0000 7f98de908640 1 -- 172.21.15.84:0/2276569681 learned_addr learned my addr 172.21.15.84:0/2276569681 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:40:24.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.836+0000 7f98dd105640 1 -- 172.21.15.84:0/2276569681 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477053446 0 0) 0x7f98d002a270 con 0x7f98d002b100 2024-03-29T10:40:24.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.836+0000 7f98dd105640 10 monclient(hunting): _negotiate 2024-03-29T10:40:24.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.836+0000 7f98dd105640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:24.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.836+0000 7f98dd105640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:24.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.836+0000 7f98dd105640 10 monclient(hunting): my global_id is 4431 2024-03-29T10:40:24.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.836+0000 7f98dd105640 1 -- 172.21.15.84:0/2276569681 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98c4003520 con 0x7f98d002b100 2024-03-29T10:40:24.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.837+0000 7f98dd105640 1 -- 172.21.15.84:0/2276569681 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1096203363 0 0) 0x7f98d00c9440 con 0x7f98d0029c30 2024-03-29T10:40:24.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.837+0000 7f98dd105640 10 monclient(hunting): _negotiate 2024-03-29T10:40:24.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.837+0000 7f98dd105640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:24.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.837+0000 7f98dd105640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:24.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.837+0000 7f98dd105640 10 monclient(hunting): my global_id is 4466 2024-03-29T10:40:24.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.837+0000 7f98dd105640 1 -- 172.21.15.84:0/2276569681 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98d002a270 con 0x7f98d0029c30 2024-03-29T10:40:24.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.837+0000 7f98dd105640 1 -- 172.21.15.84:0/2276569681 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3879936536 0 0) 0x7f98c4003520 con 0x7f98d002b100 2024-03-29T10:40:24.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.837+0000 7f98dd105640 1 -- 172.21.15.84:0/2276569681 >> v1:172.21.15.84:6790/0 conn(0x7f98d0029c30 legacy=0x7f98d00d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:24.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.837+0000 7f98dd105640 1 -- 172.21.15.84:0/2276569681 >> v1:172.21.15.86:6789/0 conn(0x7f98d0006a70 legacy=0x7f98d00c8d30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:40:24.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.838+0000 7f98dd105640 10 monclient: _finish_hunting 0 2024-03-29T10:40:24.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.838+0000 7f98dd105640 1 monclient: found mon.noname-a 2024-03-29T10:40:24.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.838+0000 7f98dd105640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:24.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.838+0000 7f98dd105640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T10:40:24.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.838+0000 7f98dd105640 1 -- 172.21.15.84:0/2276569681 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d00c95d0 con 0x7f98d002b100 2024-03-29T10:40:24.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.838+0000 7f98dd105640 10 monclient: _finish_auth 0 2024-03-29T10:40:24.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.838+0000 7f98dd105640 10 monclient: _check_auth_tickets 2024-03-29T10:40:24.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.838+0000 7f98dd105640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:24.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.838+0000 7f98dd105640 1 -- 172.21.15.84:0/2276569681 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f98c8001690 con 0x7f98d002b100 2024-03-29T10:40:24.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.838+0000 7f98dd105640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:24.823 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.838+0000 7f98dd105640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:40:24.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.838+0000 7f98dd105640 10 monclient: dump: 2024-03-29T10:40:24.824 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:24.824 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:24.824 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:24.824 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:24.824 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:24.824 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:24.824 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:24.824 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:24.824 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:24.824 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:24.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.838+0000 7f98df109640 5 monclient: authenticate success, global_id 4431 2024-03-29T10:40:24.824 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.838+0000 7f98df109640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:24.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.839+0000 7f98dd105640 1 -- 172.21.15.84:0/2276569681 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98c8001860 con 0x7f98d002b100 2024-03-29T10:40:24.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.839+0000 7f98dd105640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:24.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.839+0000 7f98df109640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:24.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.839+0000 7f98df109640 4 set_mon_vals no callback set 2024-03-29T10:40:24.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.839+0000 7f98df109640 10 monclient: shutdown 2024-03-29T10:40:24.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.840+0000 7f98df109640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:24.825 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.840+0000 7f98df109640 1 -- 172.21.15.84:0/2276569681 >> v1:172.21.15.84:6789/0 conn(0x7f98d002b100 legacy=0x7f98d00cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:24.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.840+0000 7f98dd105640 1 -- 172.21.15.84:0/2276569681 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f98c8001cb0 con 0x7f98d002b100 2024-03-29T10:40:24.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.840+0000 7f98dd105640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:40:24.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.842+0000 7f98df109640 1 -- 172.21.15.84:0/2276569681 shutdown_connections 2024-03-29T10:40:24.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.842+0000 7f98df109640 1 -- 172.21.15.84:0/2276569681 >> 172.21.15.84:0/2276569681 conn(0x7f98d00058f0 msgr2=0x7f98d0005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:24.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.842+0000 7f98df109640 1 -- 172.21.15.84:0/2276569681 shutdown_connections 2024-03-29T10:40:24.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.842+0000 7f98df109640 1 -- 172.21.15.84:0/2276569681 wait complete. 2024-03-29T10:40:24.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.843+0000 7f98df109640 10 monclient: build_initial_monmap 2024-03-29T10:40:24.828 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.844+0000 7f98df109640 10 monclient: monmap: 2024-03-29T10:40:24.828 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:40:24.828 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:24.828 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:40:24.828 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:40:24.828 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:24.828 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:24.828 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:40:24.829 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:40:24.829 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:40:24.829 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:24.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.844+0000 7f98df109640 1 librados: starting msgr at 2024-03-29T10:40:24.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.844+0000 7f98df109640 1 librados: starting objecter 2024-03-29T10:40:24.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.844+0000 7f98df109640 1 Processor -- start 2024-03-29T10:40:24.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.844+0000 7f98df109640 1 -- start start 2024-03-29T10:40:24.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.844+0000 7f98df109640 1 librados: setting wanted keys 2024-03-29T10:40:24.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.844+0000 7f98df109640 1 librados: calling monclient init 2024-03-29T10:40:24.829 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.844+0000 7f98df109640 10 monclient: init 2024-03-29T10:40:24.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.845+0000 7f98df109640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:24.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.845+0000 7f98df109640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:40:24.831 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.845+0000 7f98df109640 10 monclient(hunting): picked mon.noname-b con 0x7f98d0006a70 addr v1:172.21.15.86:6789/0 2024-03-29T10:40:24.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.846+0000 7f98df109640 10 monclient(hunting): picked mon.noname-a con 0x7f98d0029c30 addr v1:172.21.15.84:6789/0 2024-03-29T10:40:24.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.846+0000 7f98df109640 10 monclient(hunting): picked mon.noname-c con 0x7f98d0164e30 addr v1:172.21.15.84:6790/0 2024-03-29T10:40:24.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.846+0000 7f98df109640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d00caad0 con 0x7f98d0029c30 2024-03-29T10:40:24.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.846+0000 7f98dd906640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f98d0029c30 0x7f98d002da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:56812/0 (socket says 172.21.15.84:56812) 2024-03-29T10:40:24.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.846+0000 7f98dd906640 1 -- 172.21.15.84:0/1622200116 learned_addr learned my addr 172.21.15.84:0/1622200116 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:40:24.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.846+0000 7f98df109640 1 -- 172.21.15.84:0/1622200116 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d002f590 con 0x7f98d0006a70 2024-03-29T10:40:24.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.846+0000 7f98df109640 1 -- 172.21.15.84:0/1622200116 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d002cf30 con 0x7f98d0164e30 2024-03-29T10:40:24.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.846+0000 7f98df109640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:24.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.846+0000 7f98df109640 10 monclient(hunting): authenticate will time out at 1127.109741s 2024-03-29T10:40:24.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.846+0000 7f98ceffd640 10 client.?.objecter ms_handle_connect 0x7f98d0029c30 2024-03-29T10:40:24.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.846+0000 7f98ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:24.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.846+0000 7f98ceffd640 1 -- 172.21.15.84:0/1622200116 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3803195122 0 0) 0x7f98d00caad0 con 0x7f98d0029c30 2024-03-29T10:40:24.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.846+0000 7f98ceffd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:24.832 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.846+0000 7f98ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:24.833 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.846+0000 7f98ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:24.833 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.846+0000 7f98ceffd640 10 monclient(hunting): my global_id is 4434 2024-03-29T10:40:24.833 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.846+0000 7f98ceffd640 1 -- 172.21.15.84:0/1622200116 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98bc004eb0 con 0x7f98d0029c30 2024-03-29T10:40:24.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.848+0000 7f98ceffd640 10 client.?.objecter ms_handle_connect 0x7f98d0164e30 2024-03-29T10:40:24.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.848+0000 7f98ceffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:24.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.848+0000 7f98ceffd640 1 -- 172.21.15.84:0/1622200116 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1660572202 0 0) 0x7f98bc004eb0 con 0x7f98d0029c30 2024-03-29T10:40:24.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.849+0000 7f98ceffd640 1 -- 172.21.15.84:0/1622200116 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98d00caad0 con 0x7f98d0029c30 2024-03-29T10:40:24.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.849+0000 7f98ceffd640 1 -- 172.21.15.84:0/1622200116 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f98c8001c30 con 0x7f98d0029c30 2024-03-29T10:40:24.834 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.849+0000 7f98ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:24.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.849+0000 7f98ceffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:40:24.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.849+0000 7f98ceffd640 10 monclient(hunting): dump: 2024-03-29T10:40:24.835 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:24.835 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:24.835 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:24.835 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:24.835 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:24.835 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:24.835 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:24.835 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:24.835 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:24.835 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:24.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.849+0000 7f98ceffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:24.835 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.849+0000 7f98ceffd640 1 -- 172.21.15.84:0/1622200116 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2315815605 0 0) 0x7f98d002cf30 con 0x7f98d0164e30 2024-03-29T10:40:24.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.849+0000 7f98ceffd640 10 monclient(hunting): _negotiate 2024-03-29T10:40:24.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.849+0000 7f98ceffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:24.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.849+0000 7f98ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:24.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.849+0000 7f98ceffd640 10 monclient(hunting): my global_id is 4469 2024-03-29T10:40:24.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.849+0000 7f98ceffd640 1 -- 172.21.15.84:0/1622200116 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98bc004eb0 con 0x7f98d0164e30 2024-03-29T10:40:24.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.849+0000 7f98ceffd640 1 -- 172.21.15.84:0/1622200116 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1719045441 0 0) 0x7f98bc004eb0 con 0x7f98d0164e30 2024-03-29T10:40:24.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.849+0000 7f98ceffd640 1 -- 172.21.15.84:0/1622200116 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98bc005940 con 0x7f98d0164e30 2024-03-29T10:40:24.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.849+0000 7f98ceffd640 1 -- 172.21.15.84:0/1622200116 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f98d804fdd0 con 0x7f98d0164e30 2024-03-29T10:40:24.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.849+0000 7f98ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:24.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.849+0000 7f98ceffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:24.836 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.849+0000 7f98ceffd640 10 monclient(hunting): dump: 2024-03-29T10:40:24.836 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:24.836 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:24.836 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:24.836 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:24.837 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:24.837 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:24.837 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:24.837 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:24.837 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:24.837 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:24.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.850+0000 7f98ceffd640 1 -- 172.21.15.84:0/1622200116 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 33489598 0 0) 0x7f98d00caad0 con 0x7f98d0029c30 2024-03-29T10:40:24.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.850+0000 7f98ceffd640 1 -- 172.21.15.84:0/1622200116 >> v1:172.21.15.84:6790/0 conn(0x7f98d0164e30 legacy=0x7f98d016c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:24.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.850+0000 7f98ceffd640 1 -- 172.21.15.84:0/1622200116 >> v1:172.21.15.86:6789/0 conn(0x7f98d0006a70 legacy=0x7f98d0031d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:24.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.850+0000 7f98ceffd640 10 monclient: _finish_hunting 0 2024-03-29T10:40:24.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.850+0000 7f98ceffd640 1 monclient: found mon.a 2024-03-29T10:40:24.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.850+0000 7f98ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:24.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.850+0000 7f98ceffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:40:24.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.850+0000 7f98ceffd640 1 -- 172.21.15.84:0/1622200116 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d002d0c0 con 0x7f98d0029c30 2024-03-29T10:40:24.837 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.850+0000 7f98ceffd640 10 monclient: _finish_auth 0 2024-03-29T10:40:24.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.850+0000 7f98ceffd640 10 monclient: _check_auth_tickets 2024-03-29T10:40:24.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.850+0000 7f98ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:24.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.850+0000 7f98df109640 5 monclient: authenticate success, global_id 4434 2024-03-29T10:40:24.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.850+0000 7f98df109640 10 monclient: _renew_subs 2024-03-29T10:40:24.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.850+0000 7f98df109640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:40:24.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.850+0000 7f98df109640 1 -- 172.21.15.84:0/1622200116 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d0164410 con 0x7f98d0029c30 2024-03-29T10:40:24.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.850+0000 7f98df109640 10 client.4434.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:24.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.850+0000 7f98df109640 10 monclient: _renew_subs 2024-03-29T10:40:24.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.850+0000 7f98df109640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:40:24.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.850+0000 7f98df109640 1 -- 172.21.15.84:0/1622200116 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d0032410 con 0x7f98d0029c30 2024-03-29T10:40:24.838 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.850+0000 7f98df109640 1 librados: init done 2024-03-29T10:40:24.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.851+0000 7f98ceffd640 1 -- 172.21.15.84:0/1622200116 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98c80038a0 con 0x7f98d0029c30 2024-03-29T10:40:24.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.851+0000 7f98ceffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:24.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.851+0000 7f98df109640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:24.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.851+0000 7f98dc904640 4 set_mon_vals no callback set 2024-03-29T10:40:24.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.853+0000 7f98ceffd640 1 -- 172.21.15.84:0/1622200116 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f98c8003cb0 con 0x7f98d0029c30 2024-03-29T10:40:24.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.853+0000 7f98ceffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:24.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.853+0000 7f98ceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:24.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.853+0000 7f98ceffd640 10 monclient: dump: 2024-03-29T10:40:24.841 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:24.841 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:24.841 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:24.841 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:24.841 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:24.841 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:24.841 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:24.841 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:24.841 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:24.841 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:24.841 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.853+0000 7f98df109640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:24.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.853+0000 7f98df109640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:40:24.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.853+0000 7f98df109640 1 -- 172.21.15.84:0/1622200116 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d0004cd0 con 0x7f98d0029c30 2024-03-29T10:40:24.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.854+0000 7f98ceffd640 1 -- 172.21.15.84:0/1622200116 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f98c8003ed0 con 0x7f98d0029c30 2024-03-29T10:40:24.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.855+0000 7f98de107640 10 monclient: get_auth_request con 0x7f98bc06e150 auth_method 0 2024-03-29T10:40:24.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.855+0000 7f98ceffd640 1 -- 172.21.15.84:0/1622200116 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f98c8084b70 con 0x7f98d0029c30 2024-03-29T10:40:24.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.855+0000 7f98ceffd640 10 client.4434.objecter ms_dispatch 0x7f98d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:24.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.855+0000 7f98ceffd640 3 client.4434.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:24.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.855+0000 7f98ceffd640 3 client.4434.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:24.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.855+0000 7f98ceffd640 20 client.4434.objecter dump_active .. 0 homeless 2024-03-29T10:40:24.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.855+0000 7f98ceffd640 10 client.4434.objecter ms_handle_connect 0x7f98bc06e150 2024-03-29T10:40:24.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.864+0000 7f98ceffd640 1 -- 172.21.15.84:0/1622200116 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f98c8050510 con 0x7f98d0029c30 2024-03-29T10:40:24.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.864+0000 7f98ceffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:24.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:24.864+0000 7f98ceffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:25.231 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:25.248+0000 7f98df109640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:25.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:25.248+0000 7f98df109640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:25.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:25.248+0000 7f98df109640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:40:25.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:25.248+0000 7f98df109640 1 -- 172.21.15.84:0/1622200116 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98d0032760 con 0x7f98d0029c30 2024-03-29T10:40:25.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:25.248+0000 7f98ceffd640 1 -- 172.21.15.84:0/1622200116 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f98c8052440 con 0x7f98d0029c30 2024-03-29T10:40:25.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:25.248+0000 7f98ceffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:25.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:25.248+0000 7f98ceffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:25.233 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:40:25.233 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:25.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:25.252+0000 7f98df109640 10 librados: watch_flush enter 2024-03-29T10:40:25.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:25.253+0000 7f98df109640 10 librados: watch_flush exit 2024-03-29T10:40:25.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:25.253+0000 7f98df109640 20 client.4434.objecter shutdown clearing up homeless session... 2024-03-29T10:40:25.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:25.253+0000 7f98df109640 10 client.4434.objecter successfully canceled tick 2024-03-29T10:40:25.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:25.253+0000 7f98df109640 1 -- 172.21.15.84:0/1622200116 >> v1:172.21.15.86:6816/3337225337 conn(0x7f98bc06e150 legacy=0x7f98bc070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:25.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:25.253+0000 7f98df109640 10 monclient: shutdown 2024-03-29T10:40:25.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:25.253+0000 7f98df109640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:25.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:25.253+0000 7f98df109640 1 -- 172.21.15.84:0/1622200116 >> v1:172.21.15.84:6789/0 conn(0x7f98d0029c30 legacy=0x7f98d002da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:25.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:25.253+0000 7f98df109640 1 -- 172.21.15.84:0/1622200116 shutdown_connections 2024-03-29T10:40:25.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:25.253+0000 7f98df109640 1 -- 172.21.15.84:0/1622200116 >> 172.21.15.84:0/1622200116 conn(0x7f98d00058f0 msgr2=0x7f98d00c3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:25.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:25.254+0000 7f98df109640 1 -- 172.21.15.84:0/1622200116 shutdown_connections 2024-03-29T10:40:25.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:25.254+0000 7f98df109640 1 -- 172.21.15.84:0/1622200116 wait complete. 2024-03-29T10:40:25.239 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:25.254+0000 7f98df109640 1 librados: shutdown 2024-03-29T10:40:25.255 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:32.062 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:32.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.229+0000 7f55b611c640 10 monclient: get_monmap_and_config 2024-03-29T10:40:32.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.229+0000 7f55b611c640 10 monclient: build_initial_monmap 2024-03-29T10:40:32.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.230+0000 7f55b611c640 10 monclient: monmap: 2024-03-29T10:40:32.214 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:40:32.214 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:32.214 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:40:32.230937+0000 2024-03-29T10:40:32.214 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:40:32.230937+0000 2024-03-29T10:40:32.214 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:32.214 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:32.214 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:40:32.214 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:40:32.214 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:40:32.214 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:32.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.231+0000 7f55b611c640 1 Processor -- start 2024-03-29T10:40:32.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.231+0000 7f55b611c640 1 -- start start 2024-03-29T10:40:32.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.231+0000 7f55b611c640 10 monclient: init 2024-03-29T10:40:32.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.232+0000 7f55b611c640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:32.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.232+0000 7f55b611c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:40:32.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.232+0000 7f55b611c640 10 monclient(hunting): picked mon.noname-b con 0x7f55b01457f0 addr v1:172.21.15.84:6790/0 2024-03-29T10:40:32.222 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.235+0000 7f55aeffd640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f55b013f650 0x7f55b00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:49699/0 (socket says 172.21.15.84:49699) 2024-03-29T10:40:32.222 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.235+0000 7f55aeffd640 1 -- 172.21.15.84:0/1791447215 learned_addr learned my addr 172.21.15.84:0/1791447215 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:40:32.222 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.235+0000 7f55b611c640 10 monclient(hunting): picked mon.noname-a con 0x7f55b013f650 addr v1:172.21.15.84:6789/0 2024-03-29T10:40:32.222 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.235+0000 7f55b611c640 10 monclient(hunting): picked mon.noname-c con 0x7f55b00a9d70 addr v1:172.21.15.86:6789/0 2024-03-29T10:40:32.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.235+0000 7f55b611c640 1 -- 172.21.15.84:0/1791447215 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b00aaef0 con 0x7f55b013f650 2024-03-29T10:40:32.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.235+0000 7f55b611c640 1 -- 172.21.15.84:0/1791447215 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b0145bd0 con 0x7f55b00a9d70 2024-03-29T10:40:32.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.235+0000 7f55b611c640 1 -- 172.21.15.84:0/1791447215 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b0145eb0 con 0x7f55b01457f0 2024-03-29T10:40:32.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.235+0000 7f55ae7fc640 1 -- 172.21.15.84:0/1791447215 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569487817 0 0) 0x7f55b00aaef0 con 0x7f55b013f650 2024-03-29T10:40:32.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.235+0000 7f55b611c640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:32.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.235+0000 7f55b611c640 10 monclient(hunting): authenticate will time out at 1134.499093s 2024-03-29T10:40:32.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.236+0000 7f55ae7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:32.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.236+0000 7f55ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:32.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.236+0000 7f55ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:32.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.236+0000 7f55ae7fc640 10 monclient(hunting): my global_id is 4437 2024-03-29T10:40:32.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.236+0000 7f55ae7fc640 1 -- 172.21.15.84:0/1791447215 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5598003520 con 0x7f55b013f650 2024-03-29T10:40:32.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.236+0000 7f55ae7fc640 1 -- 172.21.15.84:0/1791447215 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2385491439 0 0) 0x7f55b0145eb0 con 0x7f55b01457f0 2024-03-29T10:40:32.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.236+0000 7f55ae7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:32.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.236+0000 7f55ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:32.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.236+0000 7f55ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:32.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.236+0000 7f55ae7fc640 10 monclient(hunting): my global_id is 4472 2024-03-29T10:40:32.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.236+0000 7f55ae7fc640 1 -- 172.21.15.84:0/1791447215 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55b00aaef0 con 0x7f55b01457f0 2024-03-29T10:40:32.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.236+0000 7f55ae7fc640 1 -- 172.21.15.84:0/1791447215 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3212707818 0 0) 0x7f5598003520 con 0x7f55b013f650 2024-03-29T10:40:32.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.236+0000 7f55ae7fc640 1 -- 172.21.15.84:0/1791447215 >> v1:172.21.15.84:6790/0 conn(0x7f55b01457f0 legacy=0x7f55b013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:32.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.236+0000 7f55ae7fc640 1 -- 172.21.15.84:0/1791447215 >> v1:172.21.15.86:6789/0 conn(0x7f55b00a9d70 legacy=0x7f55b014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:32.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.236+0000 7f55ae7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:40:32.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.236+0000 7f55ae7fc640 1 monclient: found mon.noname-a 2024-03-29T10:40:32.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.236+0000 7f55ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:32.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.236+0000 7f55ae7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T10:40:32.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.236+0000 7f55ae7fc640 1 -- 172.21.15.84:0/1791447215 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b0146340 con 0x7f55b013f650 2024-03-29T10:40:32.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.236+0000 7f55ae7fc640 10 monclient: _finish_auth 0 2024-03-29T10:40:32.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.236+0000 7f55ae7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:40:32.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.236+0000 7f55ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:32.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.236+0000 7f55ae7fc640 1 -- 172.21.15.84:0/1791447215 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f55a0001690 con 0x7f55b013f650 2024-03-29T10:40:32.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.236+0000 7f55ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:32.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.236+0000 7f55ae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:40:32.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.236+0000 7f55ae7fc640 10 monclient: dump: 2024-03-29T10:40:32.225 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:32.225 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:32.225 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:32.225 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:32.225 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:32.225 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:32.225 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:32.225 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:32.225 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:32.225 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:32.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.237+0000 7f55b611c640 5 monclient: authenticate success, global_id 4437 2024-03-29T10:40:32.226 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.237+0000 7f55b611c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:32.226 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.237+0000 7f55ae7fc640 1 -- 172.21.15.84:0/1791447215 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55a00018b0 con 0x7f55b013f650 2024-03-29T10:40:32.226 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.237+0000 7f55ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:32.226 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.237+0000 7f55b611c640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:32.226 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.237+0000 7f55b611c640 4 set_mon_vals no callback set 2024-03-29T10:40:32.226 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.237+0000 7f55ae7fc640 1 -- 172.21.15.84:0/1791447215 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f55a0001c40 con 0x7f55b013f650 2024-03-29T10:40:32.226 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.237+0000 7f55ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:32.226 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.237+0000 7f55ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:32.226 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.237+0000 7f55ae7fc640 10 monclient: dump: 2024-03-29T10:40:32.226 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:32.226 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:32.226 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:32.226 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:32.226 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:32.227 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:32.227 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:32.227 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:32.227 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:32.227 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:32.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.238+0000 7f55b611c640 10 monclient: shutdown 2024-03-29T10:40:32.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.238+0000 7f55b611c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:32.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.238+0000 7f55b611c640 1 -- 172.21.15.84:0/1791447215 >> v1:172.21.15.84:6789/0 conn(0x7f55b013f650 legacy=0x7f55b00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:32.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.238+0000 7f55b611c640 1 -- 172.21.15.84:0/1791447215 shutdown_connections 2024-03-29T10:40:32.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.238+0000 7f55b611c640 1 -- 172.21.15.84:0/1791447215 >> 172.21.15.84:0/1791447215 conn(0x7f55b0066a00 msgr2=0x7f55b0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:32.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.239+0000 7f55b611c640 1 -- 172.21.15.84:0/1791447215 shutdown_connections 2024-03-29T10:40:32.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.239+0000 7f55b611c640 1 -- 172.21.15.84:0/1791447215 wait complete. 2024-03-29T10:40:32.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.240+0000 7f55b611c640 10 monclient: build_initial_monmap 2024-03-29T10:40:32.232 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.240+0000 7f55b611c640 10 monclient: monmap: 2024-03-29T10:40:32.232 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:40:32.232 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:32.232 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:40:32.232 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:40:32.232 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:32.232 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:32.232 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:40:32.233 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:40:32.233 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:40:32.233 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:32.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.240+0000 7f55b611c640 1 librados: starting msgr at 2024-03-29T10:40:32.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.240+0000 7f55b611c640 1 librados: starting objecter 2024-03-29T10:40:32.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.240+0000 7f55b611c640 1 Processor -- start 2024-03-29T10:40:32.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.241+0000 7f55b611c640 1 -- start start 2024-03-29T10:40:32.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.241+0000 7f55b611c640 1 librados: setting wanted keys 2024-03-29T10:40:32.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.241+0000 7f55b611c640 1 librados: calling monclient init 2024-03-29T10:40:32.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.241+0000 7f55b611c640 10 monclient: init 2024-03-29T10:40:32.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.242+0000 7f55b611c640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:32.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.242+0000 7f55b611c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:40:32.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.244+0000 7f55b611c640 10 monclient(hunting): picked mon.noname-c con 0x7f55b00a9d70 addr v1:172.21.15.84:6790/0 2024-03-29T10:40:32.233 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.244+0000 7f55b611c640 10 monclient(hunting): picked mon.noname-b con 0x7f55b013f650 addr v1:172.21.15.86:6789/0 2024-03-29T10:40:32.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.244+0000 7f55b611c640 10 monclient(hunting): picked mon.noname-a con 0x7f55b01457f0 addr v1:172.21.15.84:6789/0 2024-03-29T10:40:32.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.244+0000 7f55b611c640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b01d8130 con 0x7f55b01457f0 2024-03-29T10:40:32.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.244+0000 7f55b611c640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b00af4c0 con 0x7f55b013f650 2024-03-29T10:40:32.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.244+0000 7f55b611c640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b01d7e00 con 0x7f55b00a9d70 2024-03-29T10:40:32.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.244+0000 7f55b611c640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:32.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.244+0000 7f55b611c640 10 monclient(hunting): authenticate will time out at 1134.507840s 2024-03-29T10:40:32.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.245+0000 7f55af7fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f55b00a9d70 0x7f55b01d76f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:47545/0 (socket says 172.21.15.84:47545) 2024-03-29T10:40:32.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.245+0000 7f55af7fe640 1 -- 172.21.15.84:0/3380696147 learned_addr learned my addr 172.21.15.84:0/3380696147 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:40:32.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.245+0000 7f5597fff640 10 client.?.objecter ms_handle_connect 0x7f55b00a9d70 2024-03-29T10:40:32.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.245+0000 7f5597fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:32.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.245+0000 7f5597fff640 10 client.?.objecter ms_handle_connect 0x7f55b013f650 2024-03-29T10:40:32.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.245+0000 7f5597fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:32.234 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.245+0000 7f5597fff640 1 -- 172.21.15.84:0/3380696147 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2332987188 0 0) 0x7f55b01d7e00 con 0x7f55b00a9d70 2024-03-29T10:40:32.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.245+0000 7f5597fff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:32.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.245+0000 7f5597fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:32.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.245+0000 7f5597fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:32.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.245+0000 7f5597fff640 10 monclient(hunting): my global_id is 4475 2024-03-29T10:40:32.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.245+0000 7f5597fff640 1 -- 172.21.15.84:0/3380696147 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f559c004eb0 con 0x7f55b00a9d70 2024-03-29T10:40:32.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.245+0000 7f5597fff640 1 -- 172.21.15.84:0/3380696147 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1078897726 0 0) 0x7f55b00af4c0 con 0x7f55b013f650 2024-03-29T10:40:32.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.245+0000 7f5597fff640 10 monclient(hunting): _negotiate 2024-03-29T10:40:32.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.245+0000 7f5597fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:32.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.245+0000 7f5597fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:32.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.245+0000 7f5597fff640 10 monclient(hunting): my global_id is 4405 2024-03-29T10:40:32.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.245+0000 7f5597fff640 1 -- 172.21.15.84:0/3380696147 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55b01d7e00 con 0x7f55b013f650 2024-03-29T10:40:32.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.245+0000 7f5597fff640 1 -- 172.21.15.84:0/3380696147 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 479247898 0 0) 0x7f559c004eb0 con 0x7f55b00a9d70 2024-03-29T10:40:32.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.245+0000 7f5597fff640 1 -- 172.21.15.84:0/3380696147 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f55b00af4c0 con 0x7f55b00a9d70 2024-03-29T10:40:32.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.246+0000 7f5597fff640 1 -- 172.21.15.84:0/3380696147 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f55a4001a20 con 0x7f55b00a9d70 2024-03-29T10:40:32.235 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.246+0000 7f5597fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:32.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.246+0000 7f5597fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:32.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.246+0000 7f5597fff640 10 monclient(hunting): dump: 2024-03-29T10:40:32.236 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:32.236 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:32.236 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:32.236 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:32.236 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:32.236 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:32.236 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:32.236 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:32.236 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:32.236 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:32.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.246+0000 7f5597fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:32.236 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.246+0000 7f5597fff640 1 -- 172.21.15.84:0/3380696147 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1334846633 0 0) 0x7f55b01d7e00 con 0x7f55b013f650 2024-03-29T10:40:32.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.246+0000 7f5597fff640 1 -- 172.21.15.84:0/3380696147 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f559c004eb0 con 0x7f55b013f650 2024-03-29T10:40:32.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.246+0000 7f5597fff640 1 -- 172.21.15.84:0/3380696147 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f55a00015d0 con 0x7f55b013f650 2024-03-29T10:40:32.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.246+0000 7f5597fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:32.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.247+0000 7f5597fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:40:32.237 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.247+0000 7f5597fff640 10 monclient(hunting): dump: 2024-03-29T10:40:32.237 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:32.237 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:32.237 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:32.237 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:32.237 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:32.237 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:32.237 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:32.237 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:32.237 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:32.238 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:32.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.247+0000 7f5597fff640 1 -- 172.21.15.84:0/3380696147 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1741884342 0 0) 0x7f55b00af4c0 con 0x7f55b00a9d70 2024-03-29T10:40:32.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.247+0000 7f5597fff640 1 -- 172.21.15.84:0/3380696147 >> v1:172.21.15.86:6789/0 conn(0x7f55b013f650 legacy=0x7f55b00ab8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:32.238 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.247+0000 7f5597fff640 1 -- 172.21.15.84:0/3380696147 >> v1:172.21.15.84:6789/0 conn(0x7f55b01457f0 legacy=0x7f55b00abff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:40:32.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.253+0000 7f5597fff640 10 monclient: _finish_hunting 0 2024-03-29T10:40:32.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.253+0000 7f5597fff640 1 monclient: found mon.c 2024-03-29T10:40:32.240 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.253+0000 7f5597fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:32.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.253+0000 7f5597fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:40:32.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.253+0000 7f5597fff640 1 -- 172.21.15.84:0/3380696147 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b0142240 con 0x7f55b00a9d70 2024-03-29T10:40:32.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.253+0000 7f5597fff640 10 monclient: _finish_auth 0 2024-03-29T10:40:32.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.253+0000 7f5597fff640 10 monclient: _check_auth_tickets 2024-03-29T10:40:32.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.253+0000 7f5597fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:32.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.253+0000 7f55b611c640 5 monclient: authenticate success, global_id 4475 2024-03-29T10:40:32.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.253+0000 7f55b611c640 10 monclient: _renew_subs 2024-03-29T10:40:32.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.253+0000 7f55b611c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:40:32.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.253+0000 7f55b611c640 1 -- 172.21.15.84:0/3380696147 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b0142490 con 0x7f55b00a9d70 2024-03-29T10:40:32.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.253+0000 7f55b611c640 10 client.4475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:32.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.253+0000 7f55b611c640 10 monclient: _renew_subs 2024-03-29T10:40:32.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.253+0000 7f55b611c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:40:32.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.253+0000 7f55b611c640 1 -- 172.21.15.84:0/3380696147 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b01e2f40 con 0x7f55b00a9d70 2024-03-29T10:40:32.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.253+0000 7f55b611c640 1 librados: init done 2024-03-29T10:40:32.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.253+0000 7f5597fff640 1 -- 172.21.15.84:0/3380696147 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55a4001cc0 con 0x7f55b00a9d70 2024-03-29T10:40:32.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.254+0000 7f5597fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:32.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.254+0000 7f55adffb640 4 set_mon_vals no callback set 2024-03-29T10:40:32.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.254+0000 7f55b611c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:32.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.255+0000 7f5597fff640 1 -- 172.21.15.84:0/3380696147 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f55a4003a60 con 0x7f55b00a9d70 2024-03-29T10:40:32.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.255+0000 7f5597fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:32.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.255+0000 7f5597fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:32.242 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.255+0000 7f5597fff640 10 monclient: dump: 2024-03-29T10:40:32.242 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:32.242 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:32.243 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:32.243 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:32.243 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:32.243 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:32.243 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:32.243 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:32.243 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:32.243 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:32.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.255+0000 7f5597fff640 1 -- 172.21.15.84:0/3380696147 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f55a4003c80 con 0x7f55b00a9d70 2024-03-29T10:40:32.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.256+0000 7f5597fff640 1 -- 172.21.15.84:0/3380696147 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f55a4084b60 con 0x7f55b00a9d70 2024-03-29T10:40:32.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.256+0000 7f5597fff640 10 client.4475.objecter ms_dispatch 0x7f55b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:32.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.256+0000 7f5597fff640 3 client.4475.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:32.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.256+0000 7f5597fff640 3 client.4475.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:32.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.256+0000 7f5597fff640 20 client.4475.objecter dump_active .. 0 homeless 2024-03-29T10:40:32.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.256+0000 7f55b611c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:32.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.256+0000 7f55b611c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:40:32.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.256+0000 7f55b611c640 1 -- 172.21.15.84:0/3380696147 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5590005080 con 0x7f55b00a9d70 2024-03-29T10:40:32.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.256+0000 7f55aeffd640 10 monclient: get_auth_request con 0x7f559c06df70 auth_method 0 2024-03-29T10:40:32.244 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.259+0000 7f5597fff640 10 client.4475.objecter ms_handle_connect 0x7f559c06df70 2024-03-29T10:40:32.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.267+0000 7f5597fff640 1 -- 172.21.15.84:0/3380696147 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f55a40504c0 con 0x7f55b00a9d70 2024-03-29T10:40:32.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.267+0000 7f5597fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:32.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.267+0000 7f5597fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:32.561 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.575+0000 7f55b611c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:32.561 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.575+0000 7f55b611c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:32.561 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.575+0000 7f55b611c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:40:32.561 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.575+0000 7f55b611c640 1 -- 172.21.15.84:0/3380696147 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5590005330 con 0x7f55b00a9d70 2024-03-29T10:40:32.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.578+0000 7f5597fff640 1 -- 172.21.15.84:0/3380696147 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f55a40523f0 con 0x7f55b00a9d70 2024-03-29T10:40:32.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.578+0000 7f5597fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:32.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.578+0000 7f5597fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:32.568 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:40:32.568 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:32.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.584+0000 7f55b611c640 10 librados: watch_flush enter 2024-03-29T10:40:32.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.584+0000 7f55b611c640 10 librados: watch_flush exit 2024-03-29T10:40:32.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.584+0000 7f55b611c640 20 client.4475.objecter shutdown clearing up homeless session... 2024-03-29T10:40:32.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.584+0000 7f55b611c640 10 client.4475.objecter successfully canceled tick 2024-03-29T10:40:32.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.585+0000 7f55b611c640 1 -- 172.21.15.84:0/3380696147 >> v1:172.21.15.86:6816/3337225337 conn(0x7f559c06df70 legacy=0x7f559c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:32.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.585+0000 7f55b611c640 10 monclient: shutdown 2024-03-29T10:40:32.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.585+0000 7f55b611c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:32.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.585+0000 7f55b611c640 1 -- 172.21.15.84:0/3380696147 >> v1:172.21.15.84:6790/0 conn(0x7f55b00a9d70 legacy=0x7f55b01d76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:32.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.585+0000 7f55b611c640 1 -- 172.21.15.84:0/3380696147 shutdown_connections 2024-03-29T10:40:32.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.585+0000 7f55b611c640 1 -- 172.21.15.84:0/3380696147 >> 172.21.15.84:0/3380696147 conn(0x7f55b0066a00 msgr2=0x7f55b0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:32.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.585+0000 7f55b611c640 1 -- 172.21.15.84:0/3380696147 shutdown_connections 2024-03-29T10:40:32.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.585+0000 7f55b611c640 1 -- 172.21.15.84:0/3380696147 wait complete. 2024-03-29T10:40:32.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:32.585+0000 7f55b611c640 1 librados: shutdown 2024-03-29T10:40:32.586 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:39.394 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:39.524 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.540+0000 7f0f4fce6640 10 monclient: get_monmap_and_config 2024-03-29T10:40:39.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.540+0000 7f0f4fce6640 10 monclient: build_initial_monmap 2024-03-29T10:40:39.525 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.540+0000 7f0f4fce6640 10 monclient: monmap: 2024-03-29T10:40:39.525 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:40:39.525 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:39.525 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:40:39.542049+0000 2024-03-29T10:40:39.525 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:40:39.542049+0000 2024-03-29T10:40:39.525 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:39.525 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:39.525 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:40:39.525 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:40:39.525 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:40:39.525 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:39.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.540+0000 7f0f4fce6640 1 Processor -- start 2024-03-29T10:40:39.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.541+0000 7f0f4fce6640 1 -- start start 2024-03-29T10:40:39.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.541+0000 7f0f4fce6640 10 monclient: init 2024-03-29T10:40:39.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.541+0000 7f0f4fce6640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:39.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.541+0000 7f0f4fce6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:40:39.527 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.541+0000 7f0f4fce6640 10 monclient(hunting): picked mon.noname-b con 0x7f0f481422b0 addr v1:172.21.15.84:6790/0 2024-03-29T10:40:39.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.541+0000 7f0f4fce6640 10 monclient(hunting): picked mon.noname-a con 0x7f0f48142690 addr v1:172.21.15.84:6789/0 2024-03-29T10:40:39.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.542+0000 7f0f4da82640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f0f481422b0 0x7f0f4813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:43834/0 (socket says 172.21.15.84:43834) 2024-03-29T10:40:39.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.542+0000 7f0f4da82640 1 -- 172.21.15.84:0/228822980 learned_addr learned my addr 172.21.15.84:0/228822980 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:40:39.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.542+0000 7f0f4fce6640 10 monclient(hunting): picked mon.noname-c con 0x7f0f480b17d0 addr v1:172.21.15.86:6789/0 2024-03-29T10:40:39.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.542+0000 7f0f4fce6640 1 -- 172.21.15.84:0/228822980 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f48142c80 con 0x7f0f48142690 2024-03-29T10:40:39.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.542+0000 7f0f4fce6640 1 -- 172.21.15.84:0/228822980 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f480aae40 con 0x7f0f480b17d0 2024-03-29T10:40:39.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.542+0000 7f0f4fce6640 1 -- 172.21.15.84:0/228822980 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f4813fbd0 con 0x7f0f481422b0 2024-03-29T10:40:39.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.542+0000 7f0f4fce6640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:39.528 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.542+0000 7f0f4fce6640 10 monclient(hunting): authenticate will time out at 1141.806603s 2024-03-29T10:40:39.529 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.544+0000 7f0f4ca80640 1 -- 172.21.15.84:0/228822980 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3185897758 0 0) 0x7f0f4813fbd0 con 0x7f0f481422b0 2024-03-29T10:40:39.529 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.544+0000 7f0f4ca80640 10 monclient(hunting): _negotiate 2024-03-29T10:40:39.529 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.544+0000 7f0f4ca80640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:39.529 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.544+0000 7f0f4ca80640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:39.529 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.544+0000 7f0f4ca80640 10 monclient(hunting): my global_id is 4478 2024-03-29T10:40:39.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.544+0000 7f0f4ca80640 1 -- 172.21.15.84:0/228822980 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f34003520 con 0x7f0f481422b0 2024-03-29T10:40:39.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.544+0000 7f0f4ca80640 1 -- 172.21.15.84:0/228822980 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2680391817 0 0) 0x7f0f34003520 con 0x7f0f481422b0 2024-03-29T10:40:39.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.544+0000 7f0f4ca80640 1 -- 172.21.15.84:0/228822980 >> v1:172.21.15.86:6789/0 conn(0x7f0f480b17d0 legacy=0x7f0f480b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:39.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.544+0000 7f0f4ca80640 1 -- 172.21.15.84:0/228822980 >> v1:172.21.15.84:6789/0 conn(0x7f0f48142690 legacy=0x7f0f480b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:40:39.531 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.545+0000 7f0f4ca80640 10 monclient: _finish_hunting 0 2024-03-29T10:40:39.531 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.545+0000 7f0f4ca80640 1 monclient: found mon.noname-b 2024-03-29T10:40:39.531 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.545+0000 7f0f4ca80640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:39.531 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.545+0000 7f0f4ca80640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:40:39.531 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.545+0000 7f0f4ca80640 1 -- 172.21.15.84:0/228822980 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f48151940 con 0x7f0f481422b0 2024-03-29T10:40:39.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.545+0000 7f0f4ca80640 10 monclient: _finish_auth 0 2024-03-29T10:40:39.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.545+0000 7f0f4ca80640 10 monclient: _check_auth_tickets 2024-03-29T10:40:39.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.545+0000 7f0f4ca80640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:39.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.545+0000 7f0f4ca80640 1 -- 172.21.15.84:0/228822980 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0f38001690 con 0x7f0f481422b0 2024-03-29T10:40:39.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.545+0000 7f0f4ca80640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:39.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.546+0000 7f0f4ca80640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:39.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.546+0000 7f0f4ca80640 10 monclient: dump: 2024-03-29T10:40:39.532 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:39.532 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:39.532 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:39.533 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:39.533 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:39.533 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:39.533 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:39.533 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:39.533 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:39.533 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:39.533 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.546+0000 7f0f4fce6640 5 monclient: authenticate success, global_id 4478 2024-03-29T10:40:39.533 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.546+0000 7f0f4fce6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:39.533 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.546+0000 7f0f4ca80640 1 -- 172.21.15.84:0/228822980 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f380018b0 con 0x7f0f481422b0 2024-03-29T10:40:39.533 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.546+0000 7f0f4ca80640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:39.533 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.546+0000 7f0f4ca80640 1 -- 172.21.15.84:0/228822980 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0f38001c40 con 0x7f0f481422b0 2024-03-29T10:40:39.533 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.546+0000 7f0f4fce6640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:39.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.546+0000 7f0f4fce6640 4 set_mon_vals no callback set 2024-03-29T10:40:39.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.546+0000 7f0f4ca80640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:39.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.546+0000 7f0f4ca80640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:39.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.546+0000 7f0f4ca80640 10 monclient: dump: 2024-03-29T10:40:39.534 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:39.534 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:39.534 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:39.534 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:39.534 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:39.534 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:39.534 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:39.534 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:39.534 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:39.534 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:39.535 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.547+0000 7f0f4fce6640 10 monclient: shutdown 2024-03-29T10:40:39.535 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.547+0000 7f0f4fce6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:39.535 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.547+0000 7f0f4fce6640 1 -- 172.21.15.84:0/228822980 >> v1:172.21.15.84:6790/0 conn(0x7f0f481422b0 legacy=0x7f0f4813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:39.536 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.547+0000 7f0f4fce6640 1 -- 172.21.15.84:0/228822980 shutdown_connections 2024-03-29T10:40:39.536 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.547+0000 7f0f4fce6640 1 -- 172.21.15.84:0/228822980 >> 172.21.15.84:0/228822980 conn(0x7f0f48066a00 msgr2=0x7f0f48066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:39.536 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.547+0000 7f0f4fce6640 1 -- 172.21.15.84:0/228822980 shutdown_connections 2024-03-29T10:40:39.536 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.547+0000 7f0f4fce6640 1 -- 172.21.15.84:0/228822980 wait complete. 2024-03-29T10:40:39.536 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.548+0000 7f0f4fce6640 10 monclient: build_initial_monmap 2024-03-29T10:40:39.536 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.548+0000 7f0f4fce6640 10 monclient: monmap: 2024-03-29T10:40:39.536 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:40:39.536 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:39.536 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:40:39.536 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:40:39.537 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:39.537 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:39.537 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:40:39.537 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:40:39.537 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:40:39.537 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:39.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.549+0000 7f0f4fce6640 1 librados: starting msgr at 2024-03-29T10:40:39.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.549+0000 7f0f4fce6640 1 librados: starting objecter 2024-03-29T10:40:39.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.549+0000 7f0f4fce6640 1 Processor -- start 2024-03-29T10:40:39.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.549+0000 7f0f4fce6640 1 -- start start 2024-03-29T10:40:39.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.549+0000 7f0f4fce6640 1 librados: setting wanted keys 2024-03-29T10:40:39.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.549+0000 7f0f4fce6640 1 librados: calling monclient init 2024-03-29T10:40:39.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.549+0000 7f0f4fce6640 10 monclient: init 2024-03-29T10:40:39.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.549+0000 7f0f4fce6640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:39.537 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.550+0000 7f0f4fce6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:40:39.540 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.552+0000 7f0f4fce6640 10 monclient(hunting): picked mon.noname-a con 0x7f0f480b17d0 addr v1:172.21.15.84:6789/0 2024-03-29T10:40:39.540 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.552+0000 7f0f4d281640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f0f481422b0 0x7f0f4812fd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:49010/0 (socket says 172.21.15.84:49010) 2024-03-29T10:40:39.540 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.552+0000 7f0f4d281640 1 -- 172.21.15.84:0/37187976 learned_addr learned my addr 172.21.15.84:0/37187976 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:40:39.540 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.552+0000 7f0f4fce6640 10 monclient(hunting): picked mon.noname-c con 0x7f0f481422b0 addr v1:172.21.15.84:6790/0 2024-03-29T10:40:39.540 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.552+0000 7f0f3e7fc640 10 client.?.objecter ms_handle_connect 0x7f0f481422b0 2024-03-29T10:40:39.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.552+0000 7f0f3e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:39.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.552+0000 7f0f4fce6640 10 monclient(hunting): picked mon.noname-b con 0x7f0f48142690 addr v1:172.21.15.86:6789/0 2024-03-29T10:40:39.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.552+0000 7f0f4fce6640 1 -- 172.21.15.84:0/37187976 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f480ac9a0 con 0x7f0f480b17d0 2024-03-29T10:40:39.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.552+0000 7f0f4fce6640 1 -- 172.21.15.84:0/37187976 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f480acc80 con 0x7f0f48142690 2024-03-29T10:40:39.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.552+0000 7f0f4fce6640 1 -- 172.21.15.84:0/37187976 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f480acf60 con 0x7f0f481422b0 2024-03-29T10:40:39.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.552+0000 7f0f4fce6640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:39.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.552+0000 7f0f4fce6640 10 monclient(hunting): authenticate will time out at 1141.817028s 2024-03-29T10:40:39.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f3e7fc640 1 -- 172.21.15.84:0/37187976 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2877189524 0 0) 0x7f0f480acf60 con 0x7f0f481422b0 2024-03-29T10:40:39.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f3e7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:39.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f3e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:39.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:39.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f3e7fc640 10 monclient(hunting): my global_id is 4481 2024-03-29T10:40:39.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f3e7fc640 1 -- 172.21.15.84:0/37187976 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f2c004eb0 con 0x7f0f481422b0 2024-03-29T10:40:39.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f3e7fc640 1 -- 172.21.15.84:0/37187976 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1894555568 0 0) 0x7f0f2c004eb0 con 0x7f0f481422b0 2024-03-29T10:40:39.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f3e7fc640 1 -- 172.21.15.84:0/37187976 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f480acf60 con 0x7f0f481422b0 2024-03-29T10:40:39.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f3e7fc640 1 -- 172.21.15.84:0/37187976 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0f440017d0 con 0x7f0f481422b0 2024-03-29T10:40:39.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f3e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:39.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f3e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:39.542 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f3e7fc640 10 monclient(hunting): dump: 2024-03-29T10:40:39.542 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:39.542 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:39.542 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:39.542 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:39.542 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:39.542 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:39.542 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:39.543 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:39.543 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:39.543 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:39.543 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f3e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:39.543 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f3e7fc640 1 -- 172.21.15.84:0/37187976 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1588678066 0 0) 0x7f0f480acf60 con 0x7f0f481422b0 2024-03-29T10:40:39.543 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f3e7fc640 1 -- 172.21.15.84:0/37187976 >> v1:172.21.15.86:6789/0 conn(0x7f0f48142690 legacy=0x7f0f48130440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:39.543 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f3e7fc640 1 -- 172.21.15.84:0/37187976 >> v1:172.21.15.84:6789/0 conn(0x7f0f480b17d0 legacy=0x7f0f480ab7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:39.543 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f3e7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:40:39.543 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f3e7fc640 1 monclient: found mon.c 2024-03-29T10:40:39.543 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f3e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:39.543 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f3e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:40:39.543 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f3e7fc640 1 -- 172.21.15.84:0/37187976 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f481e4cb0 con 0x7f0f481422b0 2024-03-29T10:40:39.543 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f3e7fc640 10 monclient: _finish_auth 0 2024-03-29T10:40:39.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f3e7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:40:39.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:39.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f4fce6640 5 monclient: authenticate success, global_id 4481 2024-03-29T10:40:39.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f4fce6640 10 monclient: _renew_subs 2024-03-29T10:40:39.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f4fce6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:40:39.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f4fce6640 1 -- 172.21.15.84:0/37187976 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f481e2c90 con 0x7f0f481422b0 2024-03-29T10:40:39.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f4fce6640 10 client.4481.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:39.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f4fce6640 10 monclient: _renew_subs 2024-03-29T10:40:39.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f4fce6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:40:39.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f4fce6640 1 -- 172.21.15.84:0/37187976 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f481e3120 con 0x7f0f481422b0 2024-03-29T10:40:39.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.553+0000 7f0f4fce6640 1 librados: init done 2024-03-29T10:40:39.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.554+0000 7f0f3e7fc640 1 -- 172.21.15.84:0/37187976 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f44001930 con 0x7f0f481422b0 2024-03-29T10:40:39.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.554+0000 7f0f3e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:39.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.554+0000 7f0f3e7fc640 1 -- 172.21.15.84:0/37187976 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0f44004a40 con 0x7f0f481422b0 2024-03-29T10:40:39.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.554+0000 7f0f3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:39.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.554+0000 7f0f3e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:39.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.554+0000 7f0f3e7fc640 10 monclient: dump: 2024-03-29T10:40:39.545 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:39.545 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:39.545 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:39.545 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:39.545 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:39.545 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:39.545 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:39.545 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:39.545 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:39.545 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:39.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.554+0000 7f0f3ffff640 4 set_mon_vals no callback set 2024-03-29T10:40:39.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.555+0000 7f0f4fce6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:39.546 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.555+0000 7f0f4fce6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:39.546 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.555+0000 7f0f4fce6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:40:39.546 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.555+0000 7f0f4fce6640 1 -- 172.21.15.84:0/37187976 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f30005080 con 0x7f0f481422b0 2024-03-29T10:40:39.547 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.557+0000 7f0f3e7fc640 1 -- 172.21.15.84:0/37187976 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f0f4401aa10 con 0x7f0f481422b0 2024-03-29T10:40:39.547 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.557+0000 7f0f3e7fc640 1 -- 172.21.15.84:0/37187976 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f0f44084a20 con 0x7f0f481422b0 2024-03-29T10:40:39.547 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.557+0000 7f0f3e7fc640 10 client.4481.objecter ms_dispatch 0x7f0f48000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:39.547 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.557+0000 7f0f3e7fc640 3 client.4481.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:39.547 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.557+0000 7f0f3e7fc640 3 client.4481.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:39.547 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.557+0000 7f0f3e7fc640 20 client.4481.objecter dump_active .. 0 homeless 2024-03-29T10:40:39.547 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.558+0000 7f0f4da82640 10 monclient: get_auth_request con 0x7f0f2c06dcf0 auth_method 0 2024-03-29T10:40:39.547 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.559+0000 7f0f3e7fc640 10 client.4481.objecter ms_handle_connect 0x7f0f2c06dcf0 2024-03-29T10:40:39.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.564+0000 7f0f3e7fc640 1 -- 172.21.15.84:0/37187976 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0f44050130 con 0x7f0f481422b0 2024-03-29T10:40:39.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.564+0000 7f0f3e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:39.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.564+0000 7f0f3e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:39.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.881+0000 7f0f4fce6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:39.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.881+0000 7f0f4fce6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:39.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.881+0000 7f0f4fce6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:40:39.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.881+0000 7f0f4fce6640 1 -- 172.21.15.84:0/37187976 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f30005330 con 0x7f0f481422b0 2024-03-29T10:40:39.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.884+0000 7f0f3e7fc640 1 -- 172.21.15.84:0/37187976 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f0f44052230 con 0x7f0f481422b0 2024-03-29T10:40:39.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.884+0000 7f0f3e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:39.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.884+0000 7f0f3e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:39.869 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:40:39.869 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:39.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.888+0000 7f0f4fce6640 10 librados: watch_flush enter 2024-03-29T10:40:39.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.888+0000 7f0f4fce6640 10 librados: watch_flush exit 2024-03-29T10:40:39.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.888+0000 7f0f4fce6640 20 client.4481.objecter shutdown clearing up homeless session... 2024-03-29T10:40:39.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.888+0000 7f0f4fce6640 10 client.4481.objecter successfully canceled tick 2024-03-29T10:40:39.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.888+0000 7f0f4fce6640 1 -- 172.21.15.84:0/37187976 >> v1:172.21.15.86:6816/3337225337 conn(0x7f0f2c06dcf0 legacy=0x7f0f2c0701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:39.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.888+0000 7f0f4fce6640 10 monclient: shutdown 2024-03-29T10:40:39.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.888+0000 7f0f4fce6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:39.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.888+0000 7f0f4fce6640 1 -- 172.21.15.84:0/37187976 >> v1:172.21.15.84:6790/0 conn(0x7f0f481422b0 legacy=0x7f0f4812fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:39.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.888+0000 7f0f4fce6640 1 -- 172.21.15.84:0/37187976 shutdown_connections 2024-03-29T10:40:39.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.888+0000 7f0f4fce6640 1 -- 172.21.15.84:0/37187976 >> 172.21.15.84:0/37187976 conn(0x7f0f48066a00 msgr2=0x7f0f4813c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:39.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.888+0000 7f0f4fce6640 1 -- 172.21.15.84:0/37187976 shutdown_connections 2024-03-29T10:40:39.877 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.889+0000 7f0f4fce6640 1 -- 172.21.15.84:0/37187976 wait complete. 2024-03-29T10:40:39.877 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:39.889+0000 7f0f4fce6640 1 librados: shutdown 2024-03-29T10:40:39.890 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:46.697 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:46.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.859+0000 7f159bbaf640 10 monclient: get_monmap_and_config 2024-03-29T10:40:46.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.859+0000 7f159bbaf640 10 monclient: build_initial_monmap 2024-03-29T10:40:46.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.859+0000 7f159bbaf640 10 monclient: monmap: 2024-03-29T10:40:46.844 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:40:46.844 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:46.844 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:40:46.860449+0000 2024-03-29T10:40:46.844 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:40:46.860449+0000 2024-03-29T10:40:46.844 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:46.845 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:46.845 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:40:46.845 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:40:46.845 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:40:46.845 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:46.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.860+0000 7f159bbaf640 1 Processor -- start 2024-03-29T10:40:46.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.860+0000 7f159bbaf640 1 -- start start 2024-03-29T10:40:46.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.860+0000 7f159bbaf640 10 monclient: init 2024-03-29T10:40:46.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.861+0000 7f159bbaf640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:46.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.861+0000 7f159bbaf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:40:46.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.861+0000 7f159bbaf640 10 monclient(hunting): picked mon.noname-a con 0x7f15940ae250 addr v1:172.21.15.84:6789/0 2024-03-29T10:40:46.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.861+0000 7f159bbaf640 10 monclient(hunting): picked mon.noname-b con 0x7f159413f970 addr v1:172.21.15.84:6790/0 2024-03-29T10:40:46.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.861+0000 7f159bbaf640 10 monclient(hunting): picked mon.noname-c con 0x7f1594142800 addr v1:172.21.15.86:6789/0 2024-03-29T10:40:46.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.861+0000 7f159bbaf640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15940afe00 con 0x7f15940ae250 2024-03-29T10:40:46.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.861+0000 7f159bbaf640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1594142c10 con 0x7f1594142800 2024-03-29T10:40:46.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.861+0000 7f159bbaf640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15940afb20 con 0x7f159413f970 2024-03-29T10:40:46.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.861+0000 7f159bbaf640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:46.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.861+0000 7f159bbaf640 10 monclient(hunting): authenticate will time out at 1149.124965s 2024-03-29T10:40:46.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.864+0000 7f159914a640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f159413f970 0x7f15941420f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:42786/0 (socket says 172.21.15.84:42786) 2024-03-29T10:40:46.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.864+0000 7f159914a640 1 -- 172.21.15.84:0/2599248503 learned_addr learned my addr 172.21.15.84:0/2599248503 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:40:46.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.865+0000 7f1598949640 1 -- 172.21.15.84:0/2599248503 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2515837791 0 0) 0x7f15940afb20 con 0x7f159413f970 2024-03-29T10:40:46.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.865+0000 7f1598949640 10 monclient(hunting): _negotiate 2024-03-29T10:40:46.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.865+0000 7f1598949640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:46.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.865+0000 7f1598949640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:46.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.865+0000 7f1598949640 10 monclient(hunting): my global_id is 4484 2024-03-29T10:40:46.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.865+0000 7f1598949640 1 -- 172.21.15.84:0/2599248503 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1580003520 con 0x7f159413f970 2024-03-29T10:40:46.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.866+0000 7f1598949640 1 -- 172.21.15.84:0/2599248503 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3533321955 0 0) 0x7f15940afe00 con 0x7f15940ae250 2024-03-29T10:40:46.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.866+0000 7f1598949640 10 monclient(hunting): _negotiate 2024-03-29T10:40:46.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.866+0000 7f1598949640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:46.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.866+0000 7f1598949640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:46.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.866+0000 7f1598949640 10 monclient(hunting): my global_id is 4440 2024-03-29T10:40:46.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.866+0000 7f1598949640 1 -- 172.21.15.84:0/2599248503 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1580003840 con 0x7f15940ae250 2024-03-29T10:40:46.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.866+0000 7f1598949640 1 -- 172.21.15.84:0/2599248503 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3164777060 0 0) 0x7f1580003520 con 0x7f159413f970 2024-03-29T10:40:46.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.866+0000 7f1598949640 1 -- 172.21.15.84:0/2599248503 >> v1:172.21.15.86:6789/0 conn(0x7f1594142800 legacy=0x7f159414da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:40:46.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.866+0000 7f1598949640 1 -- 172.21.15.84:0/2599248503 >> v1:172.21.15.84:6789/0 conn(0x7f15940ae250 legacy=0x7f159413f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:46.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.866+0000 7f1598949640 10 monclient: _finish_hunting 0 2024-03-29T10:40:46.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.866+0000 7f1598949640 1 monclient: found mon.noname-b 2024-03-29T10:40:46.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.866+0000 7f1598949640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:46.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.866+0000 7f1598949640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:40:46.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.866+0000 7f1598949640 1 -- 172.21.15.84:0/2599248503 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15940aab80 con 0x7f159413f970 2024-03-29T10:40:46.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.866+0000 7f1598949640 10 monclient: _finish_auth 0 2024-03-29T10:40:46.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.866+0000 7f1598949640 10 monclient: _check_auth_tickets 2024-03-29T10:40:46.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.866+0000 7f1598949640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:46.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.866+0000 7f1598949640 1 -- 172.21.15.84:0/2599248503 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f15840015f0 con 0x7f159413f970 2024-03-29T10:40:46.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.866+0000 7f1598949640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:46.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.866+0000 7f1598949640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:46.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.866+0000 7f1598949640 10 monclient: dump: 2024-03-29T10:40:46.853 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:46.853 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:46.853 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:46.853 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:46.853 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:46.853 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:46.853 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:46.853 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:46.853 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:46.853 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:46.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.866+0000 7f1598949640 1 -- 172.21.15.84:0/2599248503 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1584001750 con 0x7f159413f970 2024-03-29T10:40:46.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.866+0000 7f1598949640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:46.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.866+0000 7f1598949640 1 -- 172.21.15.84:0/2599248503 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f1584001ba0 con 0x7f159413f970 2024-03-29T10:40:46.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.866+0000 7f1598949640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:46.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.866+0000 7f1598949640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:46.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.866+0000 7f1598949640 10 monclient: dump: 2024-03-29T10:40:46.854 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:46.854 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:46.854 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:46.854 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:46.854 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:46.854 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:46.854 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:46.854 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:46.855 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:46.855 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:46.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.868+0000 7f159bbaf640 5 monclient: authenticate success, global_id 4484 2024-03-29T10:40:46.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.868+0000 7f159bbaf640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:46.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.868+0000 7f159bbaf640 4 set_mon_vals no callback set 2024-03-29T10:40:46.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.869+0000 7f159bbaf640 10 monclient: shutdown 2024-03-29T10:40:46.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.869+0000 7f159bbaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:46.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.869+0000 7f159bbaf640 1 -- 172.21.15.84:0/2599248503 >> v1:172.21.15.84:6790/0 conn(0x7f159413f970 legacy=0x7f15941420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:46.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.869+0000 7f159bbaf640 1 -- 172.21.15.84:0/2599248503 shutdown_connections 2024-03-29T10:40:46.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.869+0000 7f159bbaf640 1 -- 172.21.15.84:0/2599248503 >> 172.21.15.84:0/2599248503 conn(0x7f15940b35e0 msgr2=0x7f15940b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:46.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.870+0000 7f159bbaf640 1 -- 172.21.15.84:0/2599248503 shutdown_connections 2024-03-29T10:40:46.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.870+0000 7f159bbaf640 1 -- 172.21.15.84:0/2599248503 wait complete. 2024-03-29T10:40:46.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.870+0000 7f159bbaf640 10 monclient: build_initial_monmap 2024-03-29T10:40:46.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.870+0000 7f159bbaf640 10 monclient: monmap: 2024-03-29T10:40:46.856 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:40:46.856 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:46.856 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:40:46.857 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:40:46.857 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:46.857 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:46.857 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:40:46.857 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:40:46.857 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:40:46.857 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:46.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.871+0000 7f159bbaf640 1 librados: starting msgr at 2024-03-29T10:40:46.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.871+0000 7f159bbaf640 1 librados: starting objecter 2024-03-29T10:40:46.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.871+0000 7f159bbaf640 1 Processor -- start 2024-03-29T10:40:46.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.871+0000 7f159bbaf640 1 -- start start 2024-03-29T10:40:46.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.871+0000 7f159bbaf640 1 librados: setting wanted keys 2024-03-29T10:40:46.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.871+0000 7f159bbaf640 1 librados: calling monclient init 2024-03-29T10:40:46.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.871+0000 7f159bbaf640 10 monclient: init 2024-03-29T10:40:46.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.872+0000 7f159bbaf640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:46.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.872+0000 7f159bbaf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:40:46.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.872+0000 7f159bbaf640 10 monclient(hunting): picked mon.noname-a con 0x7f15940ae250 addr v1:172.21.15.84:6789/0 2024-03-29T10:40:46.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.872+0000 7f159bbaf640 10 monclient(hunting): picked mon.noname-c con 0x7f159413f970 addr v1:172.21.15.84:6790/0 2024-03-29T10:40:46.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.872+0000 7f159bbaf640 10 monclient(hunting): picked mon.noname-b con 0x7f1594142800 addr v1:172.21.15.86:6789/0 2024-03-29T10:40:46.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.872+0000 7f159bbaf640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1594143730 con 0x7f15940ae250 2024-03-29T10:40:46.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.872+0000 7f159bbaf640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15941439a0 con 0x7f1594142800 2024-03-29T10:40:46.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.872+0000 7f159bbaf640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f159413bbe0 con 0x7f159413f970 2024-03-29T10:40:46.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.872+0000 7f159bbaf640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:46.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.872+0000 7f159bbaf640 10 monclient(hunting): authenticate will time out at 1149.135586s 2024-03-29T10:40:46.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.873+0000 7f159914a640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f159413f970 0x7f1594138670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:44362/0 (socket says 172.21.15.84:44362) 2024-03-29T10:40:46.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.873+0000 7f159914a640 1 -- 172.21.15.84:0/4285944175 learned_addr learned my addr 172.21.15.84:0/4285944175 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:40:46.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.873+0000 7f158a7fc640 10 client.?.objecter ms_handle_connect 0x7f159413f970 2024-03-29T10:40:46.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.873+0000 7f158a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:46.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.873+0000 7f158a7fc640 1 -- 172.21.15.84:0/4285944175 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052406055 0 0) 0x7f159413bbe0 con 0x7f159413f970 2024-03-29T10:40:46.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.873+0000 7f158a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:40:46.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.873+0000 7f158a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:46.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.873+0000 7f158a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:46.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.873+0000 7f158a7fc640 10 monclient(hunting): my global_id is 4487 2024-03-29T10:40:46.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.873+0000 7f158a7fc640 1 -- 172.21.15.84:0/4285944175 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1578004eb0 con 0x7f159413f970 2024-03-29T10:40:46.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.873+0000 7f158a7fc640 1 -- 172.21.15.84:0/4285944175 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2747636449 0 0) 0x7f1578004eb0 con 0x7f159413f970 2024-03-29T10:40:46.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.873+0000 7f158a7fc640 1 -- 172.21.15.84:0/4285944175 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f159413bbe0 con 0x7f159413f970 2024-03-29T10:40:46.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.873+0000 7f158a7fc640 1 -- 172.21.15.84:0/4285944175 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f1584001ba0 con 0x7f159413f970 2024-03-29T10:40:46.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.873+0000 7f158a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:46.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.873+0000 7f158a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:46.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.873+0000 7f158a7fc640 10 monclient(hunting): dump: 2024-03-29T10:40:46.863 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:46.863 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:46.863 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:46.863 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:46.863 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:46.863 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:46.863 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:46.863 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:46.863 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:46.863 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:46.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.873+0000 7f158a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:46.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.874+0000 7f158a7fc640 1 -- 172.21.15.84:0/4285944175 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3464482773 0 0) 0x7f159413bbe0 con 0x7f159413f970 2024-03-29T10:40:46.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.874+0000 7f158a7fc640 1 -- 172.21.15.84:0/4285944175 >> v1:172.21.15.86:6789/0 conn(0x7f1594142800 legacy=0x7f15941e4110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:40:46.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.875+0000 7f158a7fc640 1 -- 172.21.15.84:0/4285944175 >> v1:172.21.15.84:6789/0 conn(0x7f15940ae250 legacy=0x7f159413b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:46.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.875+0000 7f158a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:40:46.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.875+0000 7f158a7fc640 1 monclient: found mon.c 2024-03-29T10:40:46.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.875+0000 7f158a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:46.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.875+0000 7f158a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:40:46.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.875+0000 7f158a7fc640 1 -- 172.21.15.84:0/4285944175 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15940ac360 con 0x7f159413f970 2024-03-29T10:40:46.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.875+0000 7f158a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:40:46.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.875+0000 7f158a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:40:46.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.875+0000 7f158a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:46.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.876+0000 7f159bbaf640 5 monclient: authenticate success, global_id 4487 2024-03-29T10:40:46.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.876+0000 7f159bbaf640 10 monclient: _renew_subs 2024-03-29T10:40:46.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.876+0000 7f159bbaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:40:46.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.876+0000 7f159bbaf640 1 -- 172.21.15.84:0/4285944175 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15940ac610 con 0x7f159413f970 2024-03-29T10:40:46.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.876+0000 7f159bbaf640 10 client.4487.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:46.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.876+0000 7f159bbaf640 10 monclient: _renew_subs 2024-03-29T10:40:46.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.876+0000 7f159bbaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:40:46.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.876+0000 7f159bbaf640 1 -- 172.21.15.84:0/4285944175 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1594138f60 con 0x7f159413f970 2024-03-29T10:40:46.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.876+0000 7f159bbaf640 1 librados: init done 2024-03-29T10:40:46.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.877+0000 7f159bbaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:46.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.877+0000 7f159bbaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:46.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.877+0000 7f159bbaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:40:46.865 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.877+0000 7f159bbaf640 1 -- 172.21.15.84:0/4285944175 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15941e3e30 con 0x7f159413f970 2024-03-29T10:40:46.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.878+0000 7f158a7fc640 1 -- 172.21.15.84:0/4285944175 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1584004a90 con 0x7f159413f970 2024-03-29T10:40:46.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.878+0000 7f158a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:46.866 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.878+0000 7f158a7fc640 1 -- 172.21.15.84:0/4285944175 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f1584004d40 con 0x7f159413f970 2024-03-29T10:40:46.867 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.878+0000 7f158a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:46.867 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.878+0000 7f158a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:46.867 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.878+0000 7f158a7fc640 10 monclient: dump: 2024-03-29T10:40:46.867 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:46.867 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:46.867 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:46.867 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:46.867 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:46.867 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:46.867 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:46.867 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:46.867 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:46.867 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:46.867 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.879+0000 7f158bfff640 4 set_mon_vals no callback set 2024-03-29T10:40:46.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.882+0000 7f158a7fc640 1 -- 172.21.15.84:0/4285944175 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f1584004f20 con 0x7f159413f970 2024-03-29T10:40:46.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.883+0000 7f159994b640 10 monclient: get_auth_request con 0x7f157806dd10 auth_method 0 2024-03-29T10:40:46.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.883+0000 7f158a7fc640 1 -- 172.21.15.84:0/4285944175 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f1584054750 con 0x7f159413f970 2024-03-29T10:40:46.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.883+0000 7f158a7fc640 10 client.4487.objecter ms_dispatch 0x7f1594000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:46.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.883+0000 7f158a7fc640 3 client.4487.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:46.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.883+0000 7f158a7fc640 3 client.4487.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:46.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.883+0000 7f158a7fc640 20 client.4487.objecter dump_active .. 0 homeless 2024-03-29T10:40:46.868 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.883+0000 7f158a7fc640 10 client.4487.objecter ms_handle_connect 0x7f157806dd10 2024-03-29T10:40:46.870 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.887+0000 7f158a7fc640 1 -- 172.21.15.84:0/4285944175 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f15840509f0 con 0x7f159413f970 2024-03-29T10:40:46.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.887+0000 7f158a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:46.871 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:46.887+0000 7f158a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:47.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:47.220+0000 7f159bbaf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:47.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:47.221+0000 7f159bbaf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:47.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:47.221+0000 7f159bbaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:40:47.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:47.221+0000 7f159bbaf640 1 -- 172.21.15.84:0/4285944175 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15941e3e30 con 0x7f159413f970 2024-03-29T10:40:47.206 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:47.222+0000 7f158a7fc640 1 -- 172.21.15.84:0/4285944175 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f15840509f0 con 0x7f159413f970 2024-03-29T10:40:47.206 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:47.222+0000 7f158a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:47.206 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:47.222+0000 7f158a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:47.209 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:40:47.209 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:47.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:47.226+0000 7f159bbaf640 10 librados: watch_flush enter 2024-03-29T10:40:47.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:47.226+0000 7f159bbaf640 10 librados: watch_flush exit 2024-03-29T10:40:47.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:47.226+0000 7f159bbaf640 20 client.4487.objecter shutdown clearing up homeless session... 2024-03-29T10:40:47.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:47.226+0000 7f159bbaf640 10 client.4487.objecter successfully canceled tick 2024-03-29T10:40:47.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:47.226+0000 7f159bbaf640 1 -- 172.21.15.84:0/4285944175 >> v1:172.21.15.86:6816/3337225337 conn(0x7f157806dd10 legacy=0x7f15780701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:47.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:47.226+0000 7f159bbaf640 10 monclient: shutdown 2024-03-29T10:40:47.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:47.226+0000 7f159bbaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:47.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:47.226+0000 7f159bbaf640 1 -- 172.21.15.84:0/4285944175 >> v1:172.21.15.84:6790/0 conn(0x7f159413f970 legacy=0x7f1594138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:47.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:47.226+0000 7f159bbaf640 1 -- 172.21.15.84:0/4285944175 shutdown_connections 2024-03-29T10:40:47.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:47.226+0000 7f159bbaf640 1 -- 172.21.15.84:0/4285944175 >> 172.21.15.84:0/4285944175 conn(0x7f15940b35e0 msgr2=0x7f15940aa100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:47.211 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:47.226+0000 7f159bbaf640 1 -- 172.21.15.84:0/4285944175 shutdown_connections 2024-03-29T10:40:47.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:47.226+0000 7f159bbaf640 1 -- 172.21.15.84:0/4285944175 wait complete. 2024-03-29T10:40:47.212 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:47.227+0000 7f159bbaf640 1 librados: shutdown 2024-03-29T10:40:47.227 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:40:54.034 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:40:54.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.203+0000 7f36dc9c3640 10 monclient: get_monmap_and_config 2024-03-29T10:40:54.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.203+0000 7f36dc9c3640 10 monclient: build_initial_monmap 2024-03-29T10:40:54.187 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.203+0000 7f36dc9c3640 10 monclient: monmap: 2024-03-29T10:40:54.187 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:40:54.187 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:54.187 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:40:54.204577+0000 2024-03-29T10:40:54.188 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:40:54.204577+0000 2024-03-29T10:40:54.188 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:54.188 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:54.188 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:40:54.188 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:40:54.188 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:40:54.188 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:54.188 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.204+0000 7f36dc9c3640 1 Processor -- start 2024-03-29T10:40:54.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.205+0000 7f36dc9c3640 1 -- start start 2024-03-29T10:40:54.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.205+0000 7f36dc9c3640 10 monclient: init 2024-03-29T10:40:54.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.205+0000 7f36dc9c3640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:54.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.205+0000 7f36dc9c3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:40:54.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.205+0000 7f36dc9c3640 10 monclient(hunting): picked mon.noname-b con 0x7f36d80a9750 addr v1:172.21.15.84:6790/0 2024-03-29T10:40:54.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.205+0000 7f36dc9c3640 10 monclient(hunting): picked mon.noname-a con 0x7f36d80add40 addr v1:172.21.15.84:6789/0 2024-03-29T10:40:54.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.205+0000 7f36dc9c3640 10 monclient(hunting): picked mon.noname-c con 0x7f36d80aad20 addr v1:172.21.15.86:6789/0 2024-03-29T10:40:54.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.205+0000 7f36dc9c3640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d8149870 con 0x7f36d80add40 2024-03-29T10:40:54.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.205+0000 7f36dc9c3640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d80abf80 con 0x7f36d80aad20 2024-03-29T10:40:54.189 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.205+0000 7f36dc9c3640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d8149ac0 con 0x7f36d80a9750 2024-03-29T10:40:54.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.205+0000 7f36dc9c3640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:54.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.205+0000 7f36dc9c3640 10 monclient(hunting): authenticate will time out at 1156.469088s 2024-03-29T10:40:54.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.206+0000 7f36d6d9d640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f36d80a9750 0x7f36d8149fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:39862/0 (socket says 172.21.15.84:39862) 2024-03-29T10:40:54.190 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.206+0000 7f36d6d9d640 1 -- 172.21.15.84:0/2879791086 learned_addr learned my addr 172.21.15.84:0/2879791086 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:40:54.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.209+0000 7f36d5d9b640 1 -- 172.21.15.84:0/2879791086 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 665020617 0 0) 0x7f36d8149ac0 con 0x7f36d80a9750 2024-03-29T10:40:54.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.209+0000 7f36d5d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:40:54.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.209+0000 7f36d5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:54.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.209+0000 7f36d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:54.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.209+0000 7f36d5d9b640 10 monclient(hunting): my global_id is 4490 2024-03-29T10:40:54.192 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.209+0000 7f36d5d9b640 1 -- 172.21.15.84:0/2879791086 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36c0003520 con 0x7f36d80a9750 2024-03-29T10:40:54.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.209+0000 7f36d5d9b640 1 -- 172.21.15.84:0/2879791086 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1837815480 0 0) 0x7f36c0003520 con 0x7f36d80a9750 2024-03-29T10:40:54.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.209+0000 7f36d5d9b640 1 -- 172.21.15.84:0/2879791086 >> v1:172.21.15.86:6789/0 conn(0x7f36d80aad20 legacy=0x7f36d8151bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:40:54.193 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.209+0000 7f36d5d9b640 1 -- 172.21.15.84:0/2879791086 >> v1:172.21.15.84:6789/0 conn(0x7f36d80add40 legacy=0x7f36d8143610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:40:54.195 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.212+0000 7f36d5d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:40:54.195 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.212+0000 7f36d5d9b640 1 monclient: found mon.noname-b 2024-03-29T10:40:54.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.212+0000 7f36d5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:54.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.212+0000 7f36d5d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:40:54.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.212+0000 7f36d5d9b640 1 -- 172.21.15.84:0/2879791086 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d814a6f0 con 0x7f36d80a9750 2024-03-29T10:40:54.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.212+0000 7f36d5d9b640 10 monclient: _finish_auth 0 2024-03-29T10:40:54.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.212+0000 7f36d5d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:40:54.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.212+0000 7f36d5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:54.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.212+0000 7f36d5d9b640 1 -- 172.21.15.84:0/2879791086 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f36c8001690 con 0x7f36d80a9750 2024-03-29T10:40:54.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.212+0000 7f36d5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:54.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.212+0000 7f36d5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:40:54.196 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.212+0000 7f36d5d9b640 10 monclient: dump: 2024-03-29T10:40:54.196 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:54.196 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:54.196 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:54.196 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:54.197 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:54.197 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:54.197 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:54.197 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:54.197 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:54.197 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:54.197 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.212+0000 7f36dc9c3640 5 monclient: authenticate success, global_id 4490 2024-03-29T10:40:54.197 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.212+0000 7f36dc9c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:40:54.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.212+0000 7f36d5d9b640 1 -- 172.21.15.84:0/2879791086 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36c8001860 con 0x7f36d80a9750 2024-03-29T10:40:54.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.212+0000 7f36d5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:54.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.212+0000 7f36d5d9b640 1 -- 172.21.15.84:0/2879791086 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f36c8001cb0 con 0x7f36d80a9750 2024-03-29T10:40:54.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.212+0000 7f36d5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:54.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.212+0000 7f36d5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:40:54.198 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.212+0000 7f36d5d9b640 10 monclient: dump: 2024-03-29T10:40:54.198 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:54.198 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:54.198 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:54.198 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:54.199 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:54.199 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:54.199 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:54.199 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:54.200 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:54.200 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:54.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.212+0000 7f36dc9c3640 10 monclient: get_monmap_and_config success 2024-03-29T10:40:54.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.212+0000 7f36dc9c3640 4 set_mon_vals no callback set 2024-03-29T10:40:54.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.213+0000 7f36dc9c3640 10 monclient: shutdown 2024-03-29T10:40:54.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.213+0000 7f36dc9c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:54.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.213+0000 7f36dc9c3640 1 -- 172.21.15.84:0/2879791086 >> v1:172.21.15.84:6790/0 conn(0x7f36d80a9750 legacy=0x7f36d8149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:54.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.213+0000 7f36dc9c3640 1 -- 172.21.15.84:0/2879791086 shutdown_connections 2024-03-29T10:40:54.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.213+0000 7f36dc9c3640 1 -- 172.21.15.84:0/2879791086 >> 172.21.15.84:0/2879791086 conn(0x7f36d80673e0 msgr2=0x7f36d80b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:54.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.213+0000 7f36dc9c3640 1 -- 172.21.15.84:0/2879791086 shutdown_connections 2024-03-29T10:40:54.200 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.213+0000 7f36dc9c3640 1 -- 172.21.15.84:0/2879791086 wait complete. 2024-03-29T10:40:54.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.215+0000 7f36dc9c3640 10 monclient: build_initial_monmap 2024-03-29T10:40:54.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.215+0000 7f36dc9c3640 10 monclient: monmap: 2024-03-29T10:40:54.201 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:40:54.201 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:40:54.201 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:40:54.201 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:40:54.201 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:40:54.201 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:54.201 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:40:54.202 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:40:54.202 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:40:54.202 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:54.202 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.215+0000 7f36dc9c3640 1 librados: starting msgr at 2024-03-29T10:40:54.202 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.215+0000 7f36dc9c3640 1 librados: starting objecter 2024-03-29T10:40:54.202 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.215+0000 7f36dc9c3640 1 Processor -- start 2024-03-29T10:40:54.202 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.216+0000 7f36dc9c3640 1 -- start start 2024-03-29T10:40:54.202 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.216+0000 7f36dc9c3640 1 librados: setting wanted keys 2024-03-29T10:40:54.202 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.216+0000 7f36dc9c3640 1 librados: calling monclient init 2024-03-29T10:40:54.202 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.216+0000 7f36dc9c3640 10 monclient: init 2024-03-29T10:40:54.202 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.216+0000 7f36dc9c3640 10 monclient: _reopen_session rank -1 2024-03-29T10:40:54.202 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.216+0000 7f36dc9c3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:40:54.203 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.216+0000 7f36dc9c3640 10 monclient(hunting): picked mon.noname-a con 0x7f36d80a9750 addr v1:172.21.15.84:6789/0 2024-03-29T10:40:54.203 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.216+0000 7f36dc9c3640 10 monclient(hunting): picked mon.noname-c con 0x7f36d80aad20 addr v1:172.21.15.84:6790/0 2024-03-29T10:40:54.203 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.216+0000 7f36dc9c3640 10 monclient(hunting): picked mon.noname-b con 0x7f36d80add40 addr v1:172.21.15.86:6789/0 2024-03-29T10:40:54.203 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.216+0000 7f36dc9c3640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d806eef0 con 0x7f36d80a9750 2024-03-29T10:40:54.203 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.216+0000 7f36dc9c3640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d81473a0 con 0x7f36d80add40 2024-03-29T10:40:54.203 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.216+0000 7f36dc9c3640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d80703b0 con 0x7f36d80aad20 2024-03-29T10:40:54.203 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.216+0000 7f36dc9c3640 10 monclient(hunting): _renew_subs 2024-03-29T10:40:54.203 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.216+0000 7f36dc9c3640 10 monclient(hunting): authenticate will time out at 1156.479996s 2024-03-29T10:40:54.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.221+0000 7f36d659c640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f36d80aad20 0x7f36d8070ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:48732/0 (socket says 172.21.15.84:48732) 2024-03-29T10:40:54.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.221+0000 7f36d659c640 1 -- 172.21.15.84:0/2891741493 learned_addr learned my addr 172.21.15.84:0/2891741493 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:40:54.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.221+0000 7f36bf7fe640 10 client.?.objecter ms_handle_connect 0x7f36d80aad20 2024-03-29T10:40:54.206 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.221+0000 7f36bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:54.206 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.221+0000 7f36bf7fe640 10 client.?.objecter ms_handle_connect 0x7f36d80a9750 2024-03-29T10:40:54.206 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.221+0000 7f36bf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:40:54.206 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.221+0000 7f36bf7fe640 1 -- 172.21.15.84:0/2891741493 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1157671709 0 0) 0x7f36d80703b0 con 0x7f36d80aad20 2024-03-29T10:40:54.206 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.221+0000 7f36bf7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:54.206 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.221+0000 7f36bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:54.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.221+0000 7f36bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:54.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.221+0000 7f36bf7fe640 10 monclient(hunting): my global_id is 4493 2024-03-29T10:40:54.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.221+0000 7f36bf7fe640 1 -- 172.21.15.84:0/2891741493 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36c4004eb0 con 0x7f36d80aad20 2024-03-29T10:40:54.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.222+0000 7f36bf7fe640 1 -- 172.21.15.84:0/2891741493 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3517858550 0 0) 0x7f36d806eef0 con 0x7f36d80a9750 2024-03-29T10:40:54.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.222+0000 7f36bf7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:40:54.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.222+0000 7f36bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:40:54.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.222+0000 7f36bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:40:54.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.222+0000 7f36bf7fe640 10 monclient(hunting): my global_id is 4443 2024-03-29T10:40:54.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.222+0000 7f36bf7fe640 1 -- 172.21.15.84:0/2891741493 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36d80703b0 con 0x7f36d80a9750 2024-03-29T10:40:54.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.222+0000 7f36bf7fe640 1 -- 172.21.15.84:0/2891741493 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1373972938 0 0) 0x7f36c4004eb0 con 0x7f36d80aad20 2024-03-29T10:40:54.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.222+0000 7f36bf7fe640 1 -- 172.21.15.84:0/2891741493 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36d806eef0 con 0x7f36d80aad20 2024-03-29T10:40:54.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.222+0000 7f36bf7fe640 1 -- 172.21.15.84:0/2891741493 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f36cc0017f0 con 0x7f36d80aad20 2024-03-29T10:40:54.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.222+0000 7f36bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:54.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.222+0000 7f36bf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:40:54.213 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.222+0000 7f36bf7fe640 10 monclient(hunting): dump: 2024-03-29T10:40:54.213 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:54.213 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:54.214 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:54.214 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:54.214 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:54.214 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:54.214 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:54.214 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:54.214 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:54.214 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:54.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.222+0000 7f36bf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:40:54.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.222+0000 7f36bf7fe640 1 -- 172.21.15.84:0/2891741493 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1571751381 0 0) 0x7f36d80703b0 con 0x7f36d80a9750 2024-03-29T10:40:54.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.222+0000 7f36bf7fe640 1 -- 172.21.15.84:0/2891741493 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36c4004eb0 con 0x7f36d80a9750 2024-03-29T10:40:54.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.222+0000 7f36bf7fe640 1 -- 172.21.15.84:0/2891741493 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f36c8001b70 con 0x7f36d80a9750 2024-03-29T10:40:54.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.222+0000 7f36bf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:54.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.222+0000 7f36bf7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:54.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.222+0000 7f36bf7fe640 10 monclient(hunting): dump: 2024-03-29T10:40:54.215 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:54.215 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:54.215 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:54.215 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:54.215 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:54.215 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:54.215 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:54.216 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:54.216 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:54.216 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:54.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.222+0000 7f36bf7fe640 1 -- 172.21.15.84:0/2891741493 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2852954667 0 0) 0x7f36c4004eb0 con 0x7f36d80a9750 2024-03-29T10:40:54.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.222+0000 7f36bf7fe640 1 -- 172.21.15.84:0/2891741493 >> v1:172.21.15.84:6790/0 conn(0x7f36d80aad20 legacy=0x7f36d8070ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:54.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.223+0000 7f36bf7fe640 1 -- 172.21.15.84:0/2891741493 >> v1:172.21.15.86:6789/0 conn(0x7f36d80add40 legacy=0x7f36d81e60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:54.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.223+0000 7f36bf7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:40:54.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.223+0000 7f36bf7fe640 1 monclient: found mon.a 2024-03-29T10:40:54.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.223+0000 7f36bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:40:54.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.223+0000 7f36bf7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:40:54.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.223+0000 7f36bf7fe640 1 -- 172.21.15.84:0/2891741493 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d806f380 con 0x7f36d80a9750 2024-03-29T10:40:54.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.223+0000 7f36bf7fe640 10 monclient: _finish_auth 0 2024-03-29T10:40:54.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.223+0000 7f36bf7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:40:54.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.223+0000 7f36bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:40:54.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.223+0000 7f36dc9c3640 5 monclient: authenticate success, global_id 4443 2024-03-29T10:40:54.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.223+0000 7f36dc9c3640 10 monclient: _renew_subs 2024-03-29T10:40:54.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.223+0000 7f36dc9c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:40:54.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.223+0000 7f36dc9c3640 1 -- 172.21.15.84:0/2891741493 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d806ea10 con 0x7f36d80a9750 2024-03-29T10:40:54.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.223+0000 7f36dc9c3640 10 client.4443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:40:54.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.223+0000 7f36dc9c3640 10 monclient: _renew_subs 2024-03-29T10:40:54.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.223+0000 7f36dc9c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:40:54.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.223+0000 7f36dc9c3640 1 -- 172.21.15.84:0/2891741493 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d80713a0 con 0x7f36d80a9750 2024-03-29T10:40:54.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.223+0000 7f36dc9c3640 1 librados: init done 2024-03-29T10:40:54.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.226+0000 7f36bf7fe640 1 -- 172.21.15.84:0/2891741493 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36c8001cd0 con 0x7f36d80a9750 2024-03-29T10:40:54.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.226+0000 7f36bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:40:54.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.226+0000 7f36bf7fe640 1 -- 172.21.15.84:0/2891741493 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f36c8004d90 con 0x7f36d80a9750 2024-03-29T10:40:54.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.226+0000 7f36d559a640 4 set_mon_vals no callback set 2024-03-29T10:40:54.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.226+0000 7f36bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:40:54.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.226+0000 7f36bf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:40:54.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.226+0000 7f36bf7fe640 10 monclient: dump: 2024-03-29T10:40:54.218 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:40:54.218 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:40:54.218 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:54.218 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:40:54.218 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:40:54.218 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:40:54.218 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:40:54.218 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:40:54.219 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:40:54.219 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:40:54.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.227+0000 7f36bf7fe640 1 -- 172.21.15.84:0/2891741493 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f36c801ad60 con 0x7f36d80a9750 2024-03-29T10:40:54.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.227+0000 7f36bf7fe640 1 -- 172.21.15.84:0/2891741493 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f36c8084a10 con 0x7f36d80a9750 2024-03-29T10:40:54.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.227+0000 7f36bf7fe640 10 client.4443.objecter ms_dispatch 0x7f36d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:40:54.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.227+0000 7f36bf7fe640 3 client.4443.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:40:54.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.227+0000 7f36bf7fe640 3 client.4443.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:40:54.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.227+0000 7f36bf7fe640 20 client.4443.objecter dump_active .. 0 homeless 2024-03-29T10:40:54.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.228+0000 7f36dc9c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:54.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.229+0000 7f36dc9c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:54.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.229+0000 7f36dc9c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:40:54.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.229+0000 7f36dc9c3640 1 -- 172.21.15.84:0/2891741493 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b8005080 con 0x7f36d80a9750 2024-03-29T10:40:54.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.229+0000 7f36d659c640 10 monclient: get_auth_request con 0x7f36c406df50 auth_method 0 2024-03-29T10:40:54.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.229+0000 7f36bf7fe640 10 client.4443.objecter ms_handle_connect 0x7f36c406df50 2024-03-29T10:40:54.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.237+0000 7f36bf7fe640 1 -- 172.21.15.84:0/2891741493 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f36c80502f0 con 0x7f36d80a9750 2024-03-29T10:40:54.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.237+0000 7f36bf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:40:54.222 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.237+0000 7f36bf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:40:54.550 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.567+0000 7f36dc9c3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:54.551 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.567+0000 7f36dc9c3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:54.551 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.567+0000 7f36dc9c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:40:54.551 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.567+0000 7f36dc9c3640 1 -- 172.21.15.84:0/2891741493 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36b8005330 con 0x7f36d80a9750 2024-03-29T10:40:54.553 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.569+0000 7f36bf7fe640 1 -- 172.21.15.84:0/2891741493 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f36c8052220 con 0x7f36d80a9750 2024-03-29T10:40:54.553 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.569+0000 7f36bf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:40:54.553 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.569+0000 7f36bf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:40:54.553 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:40:54.553 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:40:54.556 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.572+0000 7f36dc9c3640 10 librados: watch_flush enter 2024-03-29T10:40:54.556 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.573+0000 7f36dc9c3640 10 librados: watch_flush exit 2024-03-29T10:40:54.556 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.573+0000 7f36dc9c3640 20 client.4443.objecter shutdown clearing up homeless session... 2024-03-29T10:40:54.557 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.573+0000 7f36dc9c3640 10 client.4443.objecter successfully canceled tick 2024-03-29T10:40:54.557 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.573+0000 7f36dc9c3640 1 -- 172.21.15.84:0/2891741493 >> v1:172.21.15.86:6816/3337225337 conn(0x7f36c406df50 legacy=0x7f36c4070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:54.557 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.573+0000 7f36dc9c3640 10 monclient: shutdown 2024-03-29T10:40:54.557 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.573+0000 7f36dc9c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:40:54.557 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.573+0000 7f36dc9c3640 1 -- 172.21.15.84:0/2891741493 >> v1:172.21.15.84:6789/0 conn(0x7f36d80a9750 legacy=0x7f36d806e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:40:54.557 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.573+0000 7f36dc9c3640 1 -- 172.21.15.84:0/2891741493 shutdown_connections 2024-03-29T10:40:54.557 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.573+0000 7f36dc9c3640 1 -- 172.21.15.84:0/2891741493 >> 172.21.15.84:0/2891741493 conn(0x7f36d80673e0 msgr2=0x7f36d8149fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:40:54.558 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.574+0000 7f36dc9c3640 1 -- 172.21.15.84:0/2891741493 shutdown_connections 2024-03-29T10:40:54.558 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.574+0000 7f36dc9c3640 1 -- 172.21.15.84:0/2891741493 wait complete. 2024-03-29T10:40:54.558 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:40:54.574+0000 7f36dc9c3640 1 librados: shutdown 2024-03-29T10:40:54.577 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:01.385 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:01.529 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.545+0000 7f156f59e640 10 monclient: get_monmap_and_config 2024-03-29T10:41:01.529 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.545+0000 7f156f59e640 10 monclient: build_initial_monmap 2024-03-29T10:41:01.529 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.545+0000 7f156f59e640 10 monclient: monmap: 2024-03-29T10:41:01.529 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:41:01.529 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:01.529 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:41:01.546140+0000 2024-03-29T10:41:01.529 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:41:01.546140+0000 2024-03-29T10:41:01.529 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:01.529 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:01.529 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:41:01.529 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:41:01.529 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:41:01.530 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:01.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.546+0000 7f156f59e640 1 Processor -- start 2024-03-29T10:41:01.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.546+0000 7f156f59e640 1 -- start start 2024-03-29T10:41:01.530 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.546+0000 7f156f59e640 10 monclient: init 2024-03-29T10:41:01.531 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.547+0000 7f156f59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:01.531 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.547+0000 7f156f59e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:41:01.531 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.547+0000 7f156f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f15680389e0 addr v1:172.21.15.84:6789/0 2024-03-29T10:41:01.531 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.547+0000 7f156f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f1568038dc0 addr v1:172.21.15.84:6790/0 2024-03-29T10:41:01.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.547+0000 7f156f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f1568030660 addr v1:172.21.15.86:6789/0 2024-03-29T10:41:01.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.548+0000 7f156f59e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15680c54c0 con 0x7f15680389e0 2024-03-29T10:41:01.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.548+0000 7f156f59e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1568036430 con 0x7f1568030660 2024-03-29T10:41:01.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.548+0000 7f156f59e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1568039840 con 0x7f1568038dc0 2024-03-29T10:41:01.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.548+0000 7f156f59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:01.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.548+0000 7f156f59e640 10 monclient(hunting): authenticate will time out at 1163.811345s 2024-03-29T10:41:01.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.548+0000 7f156dd9b640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f1568038dc0 0x7f15680cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:46410/0 (socket says 172.21.15.84:46410) 2024-03-29T10:41:01.532 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.548+0000 7f156dd9b640 1 -- 172.21.15.84:0/3988921090 learned_addr learned my addr 172.21.15.84:0/3988921090 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:41:01.533 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.548+0000 7f156d59a640 1 -- 172.21.15.84:0/3988921090 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 74195312 0 0) 0x7f1568039840 con 0x7f1568038dc0 2024-03-29T10:41:01.533 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.549+0000 7f156d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:41:01.533 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.549+0000 7f156d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:01.533 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.549+0000 7f156d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:01.533 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.549+0000 7f156d59a640 10 monclient(hunting): my global_id is 4496 2024-03-29T10:41:01.533 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.549+0000 7f156d59a640 1 -- 172.21.15.84:0/3988921090 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1558003520 con 0x7f1568038dc0 2024-03-29T10:41:01.533 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.549+0000 7f156d59a640 1 -- 172.21.15.84:0/3988921090 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3473772652 0 0) 0x7f1558003520 con 0x7f1568038dc0 2024-03-29T10:41:01.533 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.549+0000 7f156d59a640 1 -- 172.21.15.84:0/3988921090 >> v1:172.21.15.86:6789/0 conn(0x7f1568030660 legacy=0x7f15680d3170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:41:01.534 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.550+0000 7f156d59a640 1 -- 172.21.15.84:0/3988921090 >> v1:172.21.15.84:6789/0 conn(0x7f15680389e0 legacy=0x7f1568033520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:41:01.538 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.552+0000 7f156d59a640 10 monclient: _finish_hunting 0 2024-03-29T10:41:01.538 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.552+0000 7f156d59a640 1 monclient: found mon.noname-b 2024-03-29T10:41:01.538 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.552+0000 7f156d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:01.538 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.552+0000 7f156d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:41:01.538 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.552+0000 7f156d59a640 1 -- 172.21.15.84:0/3988921090 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1568035690 con 0x7f1568038dc0 2024-03-29T10:41:01.538 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.552+0000 7f156d59a640 10 monclient: _finish_auth 0 2024-03-29T10:41:01.538 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.552+0000 7f156d59a640 10 monclient: _check_auth_tickets 2024-03-29T10:41:01.539 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.552+0000 7f156d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:01.539 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.552+0000 7f156d59a640 1 -- 172.21.15.84:0/3988921090 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f1564001690 con 0x7f1568038dc0 2024-03-29T10:41:01.539 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.552+0000 7f156d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:01.539 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.552+0000 7f156d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:01.539 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.552+0000 7f156d59a640 10 monclient: dump: 2024-03-29T10:41:01.539 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:01.539 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:01.539 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:01.539 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:01.539 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:01.539 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:01.539 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:01.539 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:01.539 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:01.540 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:01.540 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.552+0000 7f156d59a640 1 -- 172.21.15.84:0/3988921090 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1564001860 con 0x7f1568038dc0 2024-03-29T10:41:01.540 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.552+0000 7f156d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:01.540 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.552+0000 7f156d59a640 1 -- 172.21.15.84:0/3988921090 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f1564001cb0 con 0x7f1568038dc0 2024-03-29T10:41:01.540 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.552+0000 7f156d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:01.540 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.552+0000 7f156d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:01.540 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.552+0000 7f156d59a640 10 monclient: dump: 2024-03-29T10:41:01.540 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:01.540 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:01.540 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:01.540 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:01.540 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:01.540 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:01.540 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:01.540 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:01.541 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:01.541 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:01.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.553+0000 7f156f59e640 5 monclient: authenticate success, global_id 4496 2024-03-29T10:41:01.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.553+0000 7f156f59e640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:01.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.553+0000 7f156f59e640 4 set_mon_vals no callback set 2024-03-29T10:41:01.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.554+0000 7f156f59e640 10 monclient: shutdown 2024-03-29T10:41:01.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.554+0000 7f156f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:01.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.554+0000 7f156f59e640 1 -- 172.21.15.84:0/3988921090 >> v1:172.21.15.84:6790/0 conn(0x7f1568038dc0 legacy=0x7f15680cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:01.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.554+0000 7f156f59e640 1 -- 172.21.15.84:0/3988921090 shutdown_connections 2024-03-29T10:41:01.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.554+0000 7f156f59e640 1 -- 172.21.15.84:0/3988921090 >> 172.21.15.84:0/3988921090 conn(0x7f1568005c00 msgr2=0x7f1568006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:01.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.554+0000 7f156f59e640 1 -- 172.21.15.84:0/3988921090 shutdown_connections 2024-03-29T10:41:01.541 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.554+0000 7f156f59e640 1 -- 172.21.15.84:0/3988921090 wait complete. 2024-03-29T10:41:01.543 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.555+0000 7f156f59e640 10 monclient: build_initial_monmap 2024-03-29T10:41:01.543 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.556+0000 7f156f59e640 10 monclient: monmap: 2024-03-29T10:41:01.543 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:41:01.544 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:01.544 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:41:01.544 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:41:01.544 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:01.544 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:01.544 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:41:01.544 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:41:01.544 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:41:01.544 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:01.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.556+0000 7f156f59e640 1 librados: starting msgr at 2024-03-29T10:41:01.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.556+0000 7f156f59e640 1 librados: starting objecter 2024-03-29T10:41:01.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.556+0000 7f156f59e640 1 Processor -- start 2024-03-29T10:41:01.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.556+0000 7f156f59e640 1 -- start start 2024-03-29T10:41:01.544 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.556+0000 7f156f59e640 1 librados: setting wanted keys 2024-03-29T10:41:01.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.556+0000 7f156f59e640 1 librados: calling monclient init 2024-03-29T10:41:01.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.556+0000 7f156f59e640 10 monclient: init 2024-03-29T10:41:01.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.558+0000 7f156f59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:01.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.558+0000 7f156f59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:41:01.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.558+0000 7f156f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f1568030660 addr v1:172.21.15.86:6789/0 2024-03-29T10:41:01.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.558+0000 7f156f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f15680389e0 addr v1:172.21.15.84:6789/0 2024-03-29T10:41:01.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.558+0000 7f156f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f15680c82a0 addr v1:172.21.15.84:6790/0 2024-03-29T10:41:01.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.558+0000 7f156f59e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15680c7e10 con 0x7f15680389e0 2024-03-29T10:41:01.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.558+0000 7f156f59e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f156802db20 con 0x7f1568030660 2024-03-29T10:41:01.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.558+0000 7f156f59e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f156802de00 con 0x7f15680c82a0 2024-03-29T10:41:01.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.558+0000 7f156f59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:01.545 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.558+0000 7f156f59e640 10 monclient(hunting): authenticate will time out at 1163.821715s 2024-03-29T10:41:01.546 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.561+0000 7f156ed9d640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f15680c82a0 0x7f156802eca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:48236/0 (socket says 172.21.15.84:48236) 2024-03-29T10:41:01.546 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.561+0000 7f156ed9d640 1 -- 172.21.15.84:0/4254663280 learned_addr learned my addr 172.21.15.84:0/4254663280 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:41:01.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.563+0000 7f155effd640 10 client.?.objecter ms_handle_connect 0x7f1568030660 2024-03-29T10:41:01.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.563+0000 7f155effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:01.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.563+0000 7f155effd640 10 client.?.objecter ms_handle_connect 0x7f15680c82a0 2024-03-29T10:41:01.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.563+0000 7f155effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:01.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.563+0000 7f155effd640 1 -- 172.21.15.84:0/4254663280 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1762249364 0 0) 0x7f156802de00 con 0x7f15680c82a0 2024-03-29T10:41:01.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.563+0000 7f155effd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:01.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.563+0000 7f155effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:01.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.563+0000 7f155effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:01.548 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.563+0000 7f155effd640 10 monclient(hunting): my global_id is 4499 2024-03-29T10:41:01.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.563+0000 7f155effd640 1 -- 172.21.15.84:0/4254663280 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1550004eb0 con 0x7f15680c82a0 2024-03-29T10:41:01.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.564+0000 7f155effd640 1 -- 172.21.15.84:0/4254663280 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 788181233 0 0) 0x7f156802db20 con 0x7f1568030660 2024-03-29T10:41:01.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.564+0000 7f155effd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:01.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.564+0000 7f155effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:01.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.564+0000 7f155effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:01.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.564+0000 7f155effd640 10 monclient(hunting): my global_id is 4411 2024-03-29T10:41:01.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.564+0000 7f155effd640 1 -- 172.21.15.84:0/4254663280 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15500051d0 con 0x7f1568030660 2024-03-29T10:41:01.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.564+0000 7f155effd640 1 -- 172.21.15.84:0/4254663280 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3823549848 0 0) 0x7f1550004eb0 con 0x7f15680c82a0 2024-03-29T10:41:01.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.564+0000 7f155effd640 1 -- 172.21.15.84:0/4254663280 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1550005640 con 0x7f15680c82a0 2024-03-29T10:41:01.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.564+0000 7f155effd640 1 -- 172.21.15.84:0/4254663280 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f15600017b0 con 0x7f15680c82a0 2024-03-29T10:41:01.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.564+0000 7f155effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:01.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.564+0000 7f155effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:01.549 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.564+0000 7f155effd640 10 monclient(hunting): dump: 2024-03-29T10:41:01.549 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:01.550 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:01.550 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:01.550 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:01.550 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:01.550 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:01.550 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:01.550 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:01.550 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:01.550 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:01.550 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.564+0000 7f155effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:01.550 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.564+0000 7f155effd640 1 -- 172.21.15.84:0/4254663280 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 491980929 0 0) 0x7f15500051d0 con 0x7f1568030660 2024-03-29T10:41:01.550 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.564+0000 7f155effd640 1 -- 172.21.15.84:0/4254663280 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1550005ce0 con 0x7f1568030660 2024-03-29T10:41:01.550 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.564+0000 7f155effd640 1 -- 172.21.15.84:0/4254663280 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3730739507 0 0) 0x7f1550005640 con 0x7f15680c82a0 2024-03-29T10:41:01.550 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.564+0000 7f155effd640 1 -- 172.21.15.84:0/4254663280 >> v1:172.21.15.86:6789/0 conn(0x7f1568030660 legacy=0x7f15680c5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:01.551 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.564+0000 7f155effd640 1 -- 172.21.15.84:0/4254663280 >> v1:172.21.15.84:6789/0 conn(0x7f15680389e0 legacy=0x7f156802e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:01.551 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.564+0000 7f155effd640 10 monclient: _finish_hunting 0 2024-03-29T10:41:01.551 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.564+0000 7f155effd640 1 monclient: found mon.c 2024-03-29T10:41:01.551 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.564+0000 7f155effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:01.551 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.564+0000 7f155effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:41:01.551 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.564+0000 7f155effd640 1 -- 172.21.15.84:0/4254663280 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f156802e0e0 con 0x7f15680c82a0 2024-03-29T10:41:01.551 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.564+0000 7f155effd640 10 monclient: _finish_auth 0 2024-03-29T10:41:01.551 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.564+0000 7f155effd640 10 monclient: _check_auth_tickets 2024-03-29T10:41:01.551 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.564+0000 7f155effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:01.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.564+0000 7f155effd640 1 -- 172.21.15.84:0/4254663280 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1560001d80 con 0x7f15680c82a0 2024-03-29T10:41:01.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.564+0000 7f155effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:01.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.564+0000 7f156cd99640 4 set_mon_vals no callback set 2024-03-29T10:41:01.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.565+0000 7f155effd640 1 -- 172.21.15.84:0/4254663280 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f15600017b0 con 0x7f15680c82a0 2024-03-29T10:41:01.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.565+0000 7f155effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:01.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.565+0000 7f155effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:01.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.565+0000 7f155effd640 10 monclient: dump: 2024-03-29T10:41:01.554 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:01.554 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:01.554 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:01.554 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:01.555 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:01.555 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:01.555 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:01.555 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:01.555 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:01.555 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:01.555 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.565+0000 7f156f59e640 5 monclient: authenticate success, global_id 4499 2024-03-29T10:41:01.555 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.565+0000 7f156f59e640 10 monclient: _renew_subs 2024-03-29T10:41:01.555 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.565+0000 7f156f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:41:01.555 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.565+0000 7f156f59e640 1 -- 172.21.15.84:0/4254663280 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f156802f3b0 con 0x7f15680c82a0 2024-03-29T10:41:01.555 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.566+0000 7f156f59e640 10 client.4499.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:01.555 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.566+0000 7f156f59e640 10 monclient: _renew_subs 2024-03-29T10:41:01.555 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.566+0000 7f156f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:41:01.555 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.566+0000 7f156f59e640 1 -- 172.21.15.84:0/4254663280 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15680ca110 con 0x7f15680c82a0 2024-03-29T10:41:01.556 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.566+0000 7f156f59e640 1 librados: init done 2024-03-29T10:41:01.556 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.567+0000 7f155cff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:01.556 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.567+0000 7f155cff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:01.556 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.567+0000 7f155cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:41:01.556 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.567+0000 7f155cff9640 1 -- 172.21.15.84:0/4254663280 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1568007b90 con 0x7f15680c82a0 2024-03-29T10:41:01.556 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.568+0000 7f155effd640 1 -- 172.21.15.84:0/4254663280 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f156001a710 con 0x7f15680c82a0 2024-03-29T10:41:01.556 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.569+0000 7f155effd640 1 -- 172.21.15.84:0/4254663280 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f15600542a0 con 0x7f15680c82a0 2024-03-29T10:41:01.556 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.569+0000 7f155effd640 10 client.4499.objecter ms_dispatch 0x7f1570000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:01.556 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.569+0000 7f155effd640 3 client.4499.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:01.556 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.569+0000 7f155effd640 3 client.4499.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:01.556 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.569+0000 7f155effd640 20 client.4499.objecter dump_active .. 0 homeless 2024-03-29T10:41:01.556 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.569+0000 7f156e59c640 10 monclient: get_auth_request con 0x7f155006e680 auth_method 0 2024-03-29T10:41:01.556 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.570+0000 7f155effd640 10 client.4499.objecter ms_handle_connect 0x7f155006e680 2024-03-29T10:41:01.561 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.578+0000 7f155effd640 1 -- 172.21.15.84:0/4254663280 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1560050540 con 0x7f15680c82a0 2024-03-29T10:41:01.561 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.578+0000 7f155effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:01.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.578+0000 7f155effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:01.884 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.900+0000 7f155cff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:01.884 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.900+0000 7f155cff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:01.884 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.900+0000 7f155cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:41:01.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.900+0000 7f155cff9640 1 -- 172.21.15.84:0/4254663280 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1568005120 con 0x7f15680c82a0 2024-03-29T10:41:01.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.903+0000 7f155effd640 1 -- 172.21.15.84:0/4254663280 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f15600506e0 con 0x7f15680c82a0 2024-03-29T10:41:01.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.903+0000 7f155effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:01.887 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.903+0000 7f155effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:01.888 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:41:01.888 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:01.890 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.907+0000 7f155cff9640 10 librados: watch_flush enter 2024-03-29T10:41:01.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.907+0000 7f155cff9640 10 librados: watch_flush exit 2024-03-29T10:41:01.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.907+0000 7f155cff9640 20 client.4499.objecter shutdown clearing up homeless session... 2024-03-29T10:41:01.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.907+0000 7f155cff9640 10 client.4499.objecter successfully canceled tick 2024-03-29T10:41:01.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.907+0000 7f155cff9640 1 -- 172.21.15.84:0/4254663280 >> v1:172.21.15.86:6816/3337225337 conn(0x7f155006e680 legacy=0x7f1550070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:01.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.907+0000 7f155cff9640 10 monclient: shutdown 2024-03-29T10:41:01.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.907+0000 7f155cff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:01.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.907+0000 7f155cff9640 1 -- 172.21.15.84:0/4254663280 >> v1:172.21.15.84:6790/0 conn(0x7f15680c82a0 legacy=0x7f156802eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:01.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.907+0000 7f155cff9640 1 -- 172.21.15.84:0/4254663280 shutdown_connections 2024-03-29T10:41:01.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.907+0000 7f155cff9640 1 -- 172.21.15.84:0/4254663280 >> 172.21.15.84:0/4254663280 conn(0x7f1568005c00 msgr2=0x7f15680323d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:01.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.910+0000 7f155cff9640 1 -- 172.21.15.84:0/4254663280 shutdown_connections 2024-03-29T10:41:01.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.910+0000 7f155cff9640 1 -- 172.21.15.84:0/4254663280 wait complete. 2024-03-29T10:41:01.894 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:01.910+0000 7f155cff9640 1 librados: shutdown 2024-03-29T10:41:01.916 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:08.723 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:08.873 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.889+0000 7f01e1e99640 10 monclient: get_monmap_and_config 2024-03-29T10:41:08.873 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.889+0000 7f01e1e99640 10 monclient: build_initial_monmap 2024-03-29T10:41:08.873 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.889+0000 7f01e1e99640 10 monclient: monmap: 2024-03-29T10:41:08.873 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:41:08.873 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:08.873 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:41:08.890597+0000 2024-03-29T10:41:08.873 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:41:08.890597+0000 2024-03-29T10:41:08.874 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:08.874 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:08.874 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:41:08.874 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:41:08.874 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:41:08.874 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:08.874 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.890+0000 7f01e1e99640 1 Processor -- start 2024-03-29T10:41:08.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.890+0000 7f01e1e99640 1 -- start start 2024-03-29T10:41:08.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.890+0000 7f01e1e99640 10 monclient: init 2024-03-29T10:41:08.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.891+0000 7f01e1e99640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:08.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.891+0000 7f01e1e99640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:41:08.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.891+0000 7f01e1e99640 10 monclient(hunting): picked mon.noname-c con 0x7f01dc0a9750 addr v1:172.21.15.86:6789/0 2024-03-29T10:41:08.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.891+0000 7f01e1e99640 10 monclient(hunting): picked mon.noname-a con 0x7f01dc0add40 addr v1:172.21.15.84:6789/0 2024-03-29T10:41:08.876 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.891+0000 7f01e1e99640 10 monclient(hunting): picked mon.noname-b con 0x7f01dc0aad20 addr v1:172.21.15.84:6790/0 2024-03-29T10:41:08.877 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.891+0000 7f01e1e99640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01dc149870 con 0x7f01dc0add40 2024-03-29T10:41:08.877 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.891+0000 7f01e1e99640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01dc0abf80 con 0x7f01dc0a9750 2024-03-29T10:41:08.877 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.891+0000 7f01e1e99640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01dc149ac0 con 0x7f01dc0aad20 2024-03-29T10:41:08.877 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.891+0000 7f01e1e99640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:08.877 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.891+0000 7f01e1e99640 10 monclient(hunting): authenticate will time out at 1171.154852s 2024-03-29T10:41:08.877 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.892+0000 7f01e1698640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f01dc0aad20 0x7f01dc151bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:42664/0 (socket says 172.21.15.84:42664) 2024-03-29T10:41:08.877 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.892+0000 7f01e1698640 1 -- 172.21.15.84:0/1062067425 learned_addr learned my addr 172.21.15.84:0/1062067425 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:41:08.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.895+0000 7f01db7fe640 1 -- 172.21.15.84:0/1062067425 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2239182622 0 0) 0x7f01dc149ac0 con 0x7f01dc0aad20 2024-03-29T10:41:08.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.896+0000 7f01db7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:41:08.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.896+0000 7f01db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:08.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.896+0000 7f01db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:08.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.896+0000 7f01db7fe640 10 monclient(hunting): my global_id is 4502 2024-03-29T10:41:08.879 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.896+0000 7f01db7fe640 1 -- 172.21.15.84:0/1062067425 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01c8003520 con 0x7f01dc0aad20 2024-03-29T10:41:08.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.896+0000 7f01db7fe640 1 -- 172.21.15.84:0/1062067425 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1395960101 0 0) 0x7f01c8003520 con 0x7f01dc0aad20 2024-03-29T10:41:08.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.896+0000 7f01db7fe640 1 -- 172.21.15.84:0/1062067425 >> v1:172.21.15.86:6789/0 conn(0x7f01dc0a9750 legacy=0x7f01dc149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:08.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.896+0000 7f01db7fe640 1 -- 172.21.15.84:0/1062067425 >> v1:172.21.15.84:6789/0 conn(0x7f01dc0add40 legacy=0x7f01dc143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:41:08.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.896+0000 7f01db7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:41:08.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.896+0000 7f01db7fe640 1 monclient: found mon.noname-b 2024-03-29T10:41:08.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.896+0000 7f01db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:08.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.896+0000 7f01db7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:41:08.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.896+0000 7f01db7fe640 1 -- 172.21.15.84:0/1062067425 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01dc14a6f0 con 0x7f01dc0aad20 2024-03-29T10:41:08.880 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.896+0000 7f01db7fe640 10 monclient: _finish_auth 0 2024-03-29T10:41:08.881 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.896+0000 7f01db7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:41:08.881 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.896+0000 7f01db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:08.881 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.896+0000 7f01db7fe640 1 -- 172.21.15.84:0/1062067425 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f01d4001a10 con 0x7f01dc0aad20 2024-03-29T10:41:08.881 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.896+0000 7f01db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:08.881 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.896+0000 7f01db7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:08.881 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.896+0000 7f01db7fe640 10 monclient: dump: 2024-03-29T10:41:08.881 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:08.881 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:08.881 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:08.881 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:08.881 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:08.881 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:08.881 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:08.881 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:08.881 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:08.882 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:08.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.897+0000 7f01db7fe640 1 -- 172.21.15.84:0/1062067425 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01d4001ca0 con 0x7f01dc0aad20 2024-03-29T10:41:08.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.897+0000 7f01db7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:08.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.897+0000 7f01db7fe640 1 -- 172.21.15.84:0/1062067425 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f01d4001a10 con 0x7f01dc0aad20 2024-03-29T10:41:08.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.897+0000 7f01db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:08.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.897+0000 7f01db7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:08.882 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.897+0000 7f01db7fe640 10 monclient: dump: 2024-03-29T10:41:08.882 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:08.882 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:08.883 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:08.883 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:08.883 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:08.883 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:08.883 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:08.883 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:08.883 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:08.883 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:08.884 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.898+0000 7f01e1e99640 5 monclient: authenticate success, global_id 4502 2024-03-29T10:41:08.884 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.898+0000 7f01e1e99640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:08.884 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.898+0000 7f01e1e99640 4 set_mon_vals no callback set 2024-03-29T10:41:08.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.899+0000 7f01e1e99640 10 monclient: shutdown 2024-03-29T10:41:08.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.899+0000 7f01e1e99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:08.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.899+0000 7f01e1e99640 1 -- 172.21.15.84:0/1062067425 >> v1:172.21.15.84:6790/0 conn(0x7f01dc0aad20 legacy=0x7f01dc151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:08.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.900+0000 7f01e1e99640 1 -- 172.21.15.84:0/1062067425 shutdown_connections 2024-03-29T10:41:08.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.900+0000 7f01e1e99640 1 -- 172.21.15.84:0/1062067425 >> 172.21.15.84:0/1062067425 conn(0x7f01dc0673e0 msgr2=0x7f01dc0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:08.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.900+0000 7f01e1e99640 1 -- 172.21.15.84:0/1062067425 shutdown_connections 2024-03-29T10:41:08.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.900+0000 7f01e1e99640 1 -- 172.21.15.84:0/1062067425 wait complete. 2024-03-29T10:41:08.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.901+0000 7f01e1e99640 10 monclient: build_initial_monmap 2024-03-29T10:41:08.885 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.901+0000 7f01e1e99640 10 monclient: monmap: 2024-03-29T10:41:08.886 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:41:08.886 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:08.886 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:41:08.886 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:41:08.886 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:08.886 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:08.886 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:41:08.886 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:41:08.886 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:41:08.886 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:08.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.902+0000 7f01e1e99640 1 librados: starting msgr at 2024-03-29T10:41:08.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.902+0000 7f01e1e99640 1 librados: starting objecter 2024-03-29T10:41:08.886 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.902+0000 7f01e1e99640 1 Processor -- start 2024-03-29T10:41:08.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.902+0000 7f01e1e99640 1 -- start start 2024-03-29T10:41:08.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.902+0000 7f01e1e99640 1 librados: setting wanted keys 2024-03-29T10:41:08.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.902+0000 7f01e1e99640 1 librados: calling monclient init 2024-03-29T10:41:08.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.902+0000 7f01e1e99640 10 monclient: init 2024-03-29T10:41:08.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.903+0000 7f01e1e99640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:08.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.903+0000 7f01e1e99640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:41:08.888 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.903+0000 7f01e1e99640 10 monclient(hunting): picked mon.noname-c con 0x7f01dc0a9750 addr v1:172.21.15.84:6790/0 2024-03-29T10:41:08.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.903+0000 7f01e1e99640 10 monclient(hunting): picked mon.noname-a con 0x7f01dc0aad20 addr v1:172.21.15.84:6789/0 2024-03-29T10:41:08.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.903+0000 7f01e1e99640 10 monclient(hunting): picked mon.noname-b con 0x7f01dc0add40 addr v1:172.21.15.86:6789/0 2024-03-29T10:41:08.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.903+0000 7f01e1e99640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01dc1e15c0 con 0x7f01dc0aad20 2024-03-29T10:41:08.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.903+0000 7f01e1e99640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01dc0721d0 con 0x7f01dc0add40 2024-03-29T10:41:08.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.903+0000 7f01e1e99640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01dc0724b0 con 0x7f01dc0a9750 2024-03-29T10:41:08.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.903+0000 7f01e1e99640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:08.889 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.903+0000 7f01e1e99640 10 monclient(hunting): authenticate will time out at 1171.166493s 2024-03-29T10:41:08.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.906+0000 7f01e0e97640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f01dc0a9750 0x7f01dc1e3f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:43328/0 (socket says 172.21.15.84:43328) 2024-03-29T10:41:08.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.907+0000 7f01e0e97640 1 -- 172.21.15.84:0/2896447878 learned_addr learned my addr 172.21.15.84:0/2896447878 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:41:08.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.908+0000 7f01d97fa640 10 client.?.objecter ms_handle_connect 0x7f01dc0a9750 2024-03-29T10:41:08.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.908+0000 7f01d97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:08.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.911+0000 7f01d97fa640 1 -- 172.21.15.84:0/2896447878 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439773501 0 0) 0x7f01dc0724b0 con 0x7f01dc0a9750 2024-03-29T10:41:08.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.911+0000 7f01d97fa640 10 monclient(hunting): _negotiate 2024-03-29T10:41:08.895 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.911+0000 7f01d97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:08.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.911+0000 7f01d97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:08.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.911+0000 7f01d97fa640 10 monclient(hunting): my global_id is 4505 2024-03-29T10:41:08.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.911+0000 7f01d97fa640 1 -- 172.21.15.84:0/2896447878 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01bc004eb0 con 0x7f01dc0a9750 2024-03-29T10:41:08.898 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.912+0000 7f01d97fa640 1 -- 172.21.15.84:0/2896447878 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 877092734 0 0) 0x7f01bc004eb0 con 0x7f01dc0a9750 2024-03-29T10:41:08.898 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.912+0000 7f01d97fa640 1 -- 172.21.15.84:0/2896447878 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01bc005300 con 0x7f01dc0a9750 2024-03-29T10:41:08.898 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.912+0000 7f01d97fa640 1 -- 172.21.15.84:0/2896447878 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f01c40019f0 con 0x7f01dc0a9750 2024-03-29T10:41:08.898 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.912+0000 7f01d97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:08.898 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.912+0000 7f01d97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:08.898 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.912+0000 7f01d97fa640 10 monclient(hunting): dump: 2024-03-29T10:41:08.898 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:08.899 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:08.899 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:08.899 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:08.899 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:08.899 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:08.899 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:08.899 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:08.899 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:08.899 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:08.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.912+0000 7f01d97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:08.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.912+0000 7f01d97fa640 1 -- 172.21.15.84:0/2896447878 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1335551184 0 0) 0x7f01bc005300 con 0x7f01dc0a9750 2024-03-29T10:41:08.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.912+0000 7f01d97fa640 1 -- 172.21.15.84:0/2896447878 >> v1:172.21.15.86:6789/0 conn(0x7f01dc0add40 legacy=0x7f01dc1e0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:08.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.912+0000 7f01d97fa640 1 -- 172.21.15.84:0/2896447878 >> v1:172.21.15.84:6789/0 conn(0x7f01dc0aad20 legacy=0x7f01dc1e07a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:41:08.899 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.913+0000 7f01d97fa640 10 monclient: _finish_hunting 0 2024-03-29T10:41:08.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.913+0000 7f01d97fa640 1 monclient: found mon.c 2024-03-29T10:41:08.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.913+0000 7f01d97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:08.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.913+0000 7f01d97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:41:08.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.913+0000 7f01d97fa640 1 -- 172.21.15.84:0/2896447878 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01dc1e3a50 con 0x7f01dc0a9750 2024-03-29T10:41:08.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.913+0000 7f01d97fa640 10 monclient: _finish_auth 0 2024-03-29T10:41:08.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.913+0000 7f01d97fa640 10 monclient: _check_auth_tickets 2024-03-29T10:41:08.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.913+0000 7f01d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:08.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.914+0000 7f01d97fa640 1 -- 172.21.15.84:0/2896447878 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01c4001cf0 con 0x7f01dc0a9750 2024-03-29T10:41:08.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.914+0000 7f01d97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:08.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.914+0000 7f01d97fa640 1 -- 172.21.15.84:0/2896447878 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f01c40019f0 con 0x7f01dc0a9750 2024-03-29T10:41:08.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.914+0000 7f01d97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:08.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.914+0000 7f01d97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:08.900 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.914+0000 7f01d97fa640 10 monclient: dump: 2024-03-29T10:41:08.900 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:08.901 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:08.901 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:08.901 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:08.901 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:08.901 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:08.901 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:08.901 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:08.901 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:08.901 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:08.901 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.914+0000 7f01daffd640 4 set_mon_vals no callback set 2024-03-29T10:41:08.902 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.915+0000 7f01e1e99640 5 monclient: authenticate success, global_id 4505 2024-03-29T10:41:08.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.915+0000 7f01e1e99640 10 monclient: _renew_subs 2024-03-29T10:41:08.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.915+0000 7f01e1e99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:41:08.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.915+0000 7f01e1e99640 1 -- 172.21.15.84:0/2896447878 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01dc071630 con 0x7f01dc0a9750 2024-03-29T10:41:08.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.915+0000 7f01e1e99640 10 client.4505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:08.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.915+0000 7f01e1e99640 10 monclient: _renew_subs 2024-03-29T10:41:08.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.915+0000 7f01e1e99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:41:08.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.915+0000 7f01e1e99640 1 -- 172.21.15.84:0/2896447878 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f01dc070130 con 0x7f01dc0a9750 2024-03-29T10:41:08.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.915+0000 7f01e1e99640 1 librados: init done 2024-03-29T10:41:08.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.917+0000 7f01e1e99640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:08.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.917+0000 7f01e1e99640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:08.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.917+0000 7f01e1e99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:41:08.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.917+0000 7f01e1e99640 1 -- 172.21.15.84:0/2896447878 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01dc04f730 con 0x7f01dc0a9750 2024-03-29T10:41:08.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.919+0000 7f01d97fa640 1 -- 172.21.15.84:0/2896447878 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f01c40038c0 con 0x7f01dc0a9750 2024-03-29T10:41:08.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.920+0000 7f01d97fa640 1 -- 172.21.15.84:0/2896447878 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f01c40848e0 con 0x7f01dc0a9750 2024-03-29T10:41:08.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.920+0000 7f01d97fa640 10 client.4505.objecter ms_dispatch 0x7f01dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:08.905 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.920+0000 7f01d97fa640 3 client.4505.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:08.905 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.920+0000 7f01d97fa640 3 client.4505.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:08.905 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.920+0000 7f01d97fa640 20 client.4505.objecter dump_active .. 0 homeless 2024-03-29T10:41:08.905 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.921+0000 7f01dbfff640 10 monclient: get_auth_request con 0x7f01bc06e0d0 auth_method 0 2024-03-29T10:41:08.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.925+0000 7f01d97fa640 10 client.4505.objecter ms_handle_connect 0x7f01bc06e0d0 2024-03-29T10:41:08.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.929+0000 7f01d97fa640 1 -- 172.21.15.84:0/2896447878 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f01c404fff0 con 0x7f01dc0a9750 2024-03-29T10:41:08.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.929+0000 7f01d97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:08.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:08.929+0000 7f01d97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:09.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:09.308+0000 7f01e1e99640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:09.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:09.308+0000 7f01e1e99640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:09.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:09.308+0000 7f01e1e99640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:41:09.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:09.308+0000 7f01e1e99640 1 -- 172.21.15.84:0/2896447878 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01dc1e3be0 con 0x7f01dc0a9750 2024-03-29T10:41:09.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:09.313+0000 7f01d97fa640 1 -- 172.21.15.84:0/2896447878 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f01c40520f0 con 0x7f01dc0a9750 2024-03-29T10:41:09.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:09.313+0000 7f01d97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:09.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:09.313+0000 7f01d97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:09.298 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:41:09.298 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:09.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:09.319+0000 7f01e1e99640 10 librados: watch_flush enter 2024-03-29T10:41:09.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:09.319+0000 7f01e1e99640 10 librados: watch_flush exit 2024-03-29T10:41:09.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:09.319+0000 7f01e1e99640 20 client.4505.objecter shutdown clearing up homeless session... 2024-03-29T10:41:09.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:09.319+0000 7f01e1e99640 10 client.4505.objecter successfully canceled tick 2024-03-29T10:41:09.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:09.319+0000 7f01e1e99640 1 -- 172.21.15.84:0/2896447878 >> v1:172.21.15.86:6816/3337225337 conn(0x7f01bc06e0d0 legacy=0x7f01bc070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:09.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:09.319+0000 7f01e1e99640 10 monclient: shutdown 2024-03-29T10:41:09.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:09.319+0000 7f01e1e99640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:09.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:09.319+0000 7f01e1e99640 1 -- 172.21.15.84:0/2896447878 >> v1:172.21.15.84:6790/0 conn(0x7f01dc0a9750 legacy=0x7f01dc1e3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:09.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:09.320+0000 7f01e1e99640 1 -- 172.21.15.84:0/2896447878 shutdown_connections 2024-03-29T10:41:09.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:09.320+0000 7f01e1e99640 1 -- 172.21.15.84:0/2896447878 >> 172.21.15.84:0/2896447878 conn(0x7f01dc0673e0 msgr2=0x7f01dc0aa090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:09.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:09.320+0000 7f01e1e99640 1 -- 172.21.15.84:0/2896447878 shutdown_connections 2024-03-29T10:41:09.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:09.320+0000 7f01e1e99640 1 -- 172.21.15.84:0/2896447878 wait complete. 2024-03-29T10:41:09.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:09.320+0000 7f01e1e99640 1 librados: shutdown 2024-03-29T10:41:09.322 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:16.129 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:16.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.292+0000 7f6fc24c0640 10 monclient: get_monmap_and_config 2024-03-29T10:41:16.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.292+0000 7f6fc24c0640 10 monclient: build_initial_monmap 2024-03-29T10:41:16.276 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.292+0000 7f6fc24c0640 10 monclient: monmap: 2024-03-29T10:41:16.276 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:41:16.276 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:16.277 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:41:16.293747+0000 2024-03-29T10:41:16.277 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:41:16.293747+0000 2024-03-29T10:41:16.277 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:16.277 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:16.277 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:41:16.277 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:41:16.277 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:41:16.277 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:16.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.293+0000 7f6fc24c0640 1 Processor -- start 2024-03-29T10:41:16.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.293+0000 7f6fc24c0640 1 -- start start 2024-03-29T10:41:16.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.293+0000 7f6fc24c0640 10 monclient: init 2024-03-29T10:41:16.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.294+0000 7f6fc24c0640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:16.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.294+0000 7f6fc24c0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:41:16.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.294+0000 7f6fc24c0640 10 monclient(hunting): picked mon.noname-a con 0x7f6fbc1457f0 addr v1:172.21.15.84:6789/0 2024-03-29T10:41:16.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.294+0000 7f6fc24c0640 10 monclient(hunting): picked mon.noname-b con 0x7f6fbc13f650 addr v1:172.21.15.84:6790/0 2024-03-29T10:41:16.279 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.294+0000 7f6fbb7fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f6fbc13f650 0x7f6fbc0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:56682/0 (socket says 172.21.15.84:56682) 2024-03-29T10:41:16.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.294+0000 7f6fc24c0640 10 monclient(hunting): picked mon.noname-c con 0x7f6fbc0a9d70 addr v1:172.21.15.86:6789/0 2024-03-29T10:41:16.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.294+0000 7f6fc24c0640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fbc0aaef0 con 0x7f6fbc1457f0 2024-03-29T10:41:16.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.294+0000 7f6fc24c0640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fbc145bd0 con 0x7f6fbc0a9d70 2024-03-29T10:41:16.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.295+0000 7f6fbbfff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f6fbc1457f0 0x7f6fbc13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:40914/0 (socket says 172.21.15.84:40914) 2024-03-29T10:41:16.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.295+0000 7f6fbbfff640 1 -- 172.21.15.84:0/1750676907 learned_addr learned my addr 172.21.15.84:0/1750676907 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:41:16.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.295+0000 7f6fc24c0640 1 -- 172.21.15.84:0/1750676907 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fbc145eb0 con 0x7f6fbc13f650 2024-03-29T10:41:16.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.295+0000 7f6fc24c0640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:16.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.295+0000 7f6fc24c0640 10 monclient(hunting): authenticate will time out at 1178.558372s 2024-03-29T10:41:16.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 1 -- 172.21.15.84:0/1750676907 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2780170822 0 0) 0x7f6fbc145bd0 con 0x7f6fbc0a9d70 2024-03-29T10:41:16.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:16.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:16.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:16.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 10 monclient(hunting): my global_id is 4414 2024-03-29T10:41:16.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 1 -- 172.21.15.84:0/1750676907 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fa4003520 con 0x7f6fbc0a9d70 2024-03-29T10:41:16.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 1 -- 172.21.15.84:0/1750676907 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3266285497 0 0) 0x7f6fbc0aaef0 con 0x7f6fbc1457f0 2024-03-29T10:41:16.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:16.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:16.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:16.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 10 monclient(hunting): my global_id is 4446 2024-03-29T10:41:16.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 1 -- 172.21.15.84:0/1750676907 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fbc145bd0 con 0x7f6fbc1457f0 2024-03-29T10:41:16.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 1 -- 172.21.15.84:0/1750676907 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 107980632 0 0) 0x7f6fa4003520 con 0x7f6fbc0a9d70 2024-03-29T10:41:16.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 1 -- 172.21.15.84:0/1750676907 >> v1:172.21.15.84:6790/0 conn(0x7f6fbc13f650 legacy=0x7f6fbc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:16.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 1 -- 172.21.15.84:0/1750676907 >> v1:172.21.15.84:6789/0 conn(0x7f6fbc1457f0 legacy=0x7f6fbc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:16.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 10 monclient: _finish_hunting 0 2024-03-29T10:41:16.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 1 monclient: found mon.noname-c 2024-03-29T10:41:16.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:16.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-29T10:41:16.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 1 -- 172.21.15.84:0/1750676907 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fbc146340 con 0x7f6fbc0a9d70 2024-03-29T10:41:16.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 10 monclient: _finish_auth 0 2024-03-29T10:41:16.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 10 monclient: _check_auth_tickets 2024-03-29T10:41:16.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:16.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 1 -- 172.21.15.84:0/1750676907 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f6fb4001a10 con 0x7f6fbc0a9d70 2024-03-29T10:41:16.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fc24c0640 5 monclient: authenticate success, global_id 4414 2024-03-29T10:41:16.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fc24c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:16.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:16.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:16.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 10 monclient: dump: 2024-03-29T10:41:16.284 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:16.284 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:16.284 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:16.284 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:16.284 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:16.284 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:16.285 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:16.285 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:16.285 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:16.285 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:16.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fc24c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:16.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 1 -- 172.21.15.84:0/1750676907 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fb4001c30 con 0x7f6fbc0a9d70 2024-03-29T10:41:16.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:16.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 1 -- 172.21.15.84:0/1750676907 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f6fb4001fc0 con 0x7f6fbc0a9d70 2024-03-29T10:41:16.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fc24c0640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:16.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fc24c0640 4 set_mon_vals no callback set 2024-03-29T10:41:16.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:16.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:16.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.296+0000 7f6fbaffd640 10 monclient: dump: 2024-03-29T10:41:16.286 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:16.286 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:16.286 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:16.286 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:16.286 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:16.286 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:16.287 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:16.287 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:16.287 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:16.287 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:16.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.297+0000 7f6fc24c0640 10 monclient: shutdown 2024-03-29T10:41:16.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.297+0000 7f6fc24c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:16.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.297+0000 7f6fc24c0640 1 -- 172.21.15.84:0/1750676907 >> v1:172.21.15.86:6789/0 conn(0x7f6fbc0a9d70 legacy=0x7f6fbc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:16.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.297+0000 7f6fc24c0640 1 -- 172.21.15.84:0/1750676907 shutdown_connections 2024-03-29T10:41:16.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.297+0000 7f6fc24c0640 1 -- 172.21.15.84:0/1750676907 >> 172.21.15.84:0/1750676907 conn(0x7f6fbc066a00 msgr2=0x7f6fbc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:16.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.298+0000 7f6fc24c0640 1 -- 172.21.15.84:0/1750676907 shutdown_connections 2024-03-29T10:41:16.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.298+0000 7f6fc24c0640 1 -- 172.21.15.84:0/1750676907 wait complete. 2024-03-29T10:41:16.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.299+0000 7f6fc24c0640 10 monclient: build_initial_monmap 2024-03-29T10:41:16.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.299+0000 7f6fc24c0640 10 monclient: monmap: 2024-03-29T10:41:16.300 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:41:16.300 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:16.300 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:41:16.300 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:41:16.300 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:16.300 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:16.300 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:41:16.300 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:41:16.301 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:41:16.301 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:16.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.299+0000 7f6fc24c0640 1 librados: starting msgr at 2024-03-29T10:41:16.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.299+0000 7f6fc24c0640 1 librados: starting objecter 2024-03-29T10:41:16.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.300+0000 7f6fc24c0640 1 Processor -- start 2024-03-29T10:41:16.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.300+0000 7f6fc24c0640 1 -- start start 2024-03-29T10:41:16.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.300+0000 7f6fc24c0640 1 librados: setting wanted keys 2024-03-29T10:41:16.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.300+0000 7f6fc24c0640 1 librados: calling monclient init 2024-03-29T10:41:16.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.300+0000 7f6fc24c0640 10 monclient: init 2024-03-29T10:41:16.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.301+0000 7f6fc24c0640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:16.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.301+0000 7f6fc24c0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:41:16.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.301+0000 7f6fc24c0640 10 monclient(hunting): picked mon.noname-b con 0x7f6fbc0a9d70 addr v1:172.21.15.86:6789/0 2024-03-29T10:41:16.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.301+0000 7f6fc24c0640 10 monclient(hunting): picked mon.noname-a con 0x7f6fbc13f650 addr v1:172.21.15.84:6789/0 2024-03-29T10:41:16.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.301+0000 7f6fbb7fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f6fbc13f650 0x7f6fbc136970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:41314/0 (socket says 172.21.15.84:41314) 2024-03-29T10:41:16.302 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.301+0000 7f6fbb7fe640 1 -- 172.21.15.84:0/3220601363 learned_addr learned my addr 172.21.15.84:0/3220601363 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:41:16.302 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.302+0000 7f6fb8ff9640 10 client.?.objecter ms_handle_connect 0x7f6fbc13f650 2024-03-29T10:41:16.302 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.302+0000 7f6fc24c0640 10 monclient(hunting): picked mon.noname-c con 0x7f6fbc1457f0 addr v1:172.21.15.84:6790/0 2024-03-29T10:41:16.302 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.302+0000 7f6fc24c0640 1 -- 172.21.15.84:0/3220601363 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fbc140410 con 0x7f6fbc13f650 2024-03-29T10:41:16.302 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.302+0000 7f6fc24c0640 1 -- 172.21.15.84:0/3220601363 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fbc06eb20 con 0x7f6fbc0a9d70 2024-03-29T10:41:16.302 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.302+0000 7f6fc24c0640 1 -- 172.21.15.84:0/3220601363 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fbc141f00 con 0x7f6fbc1457f0 2024-03-29T10:41:16.302 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.302+0000 7f6fc24c0640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:16.302 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.302+0000 7f6fc24c0640 10 monclient(hunting): authenticate will time out at 1178.565504s 2024-03-29T10:41:16.302 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.302+0000 7f6fb8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:16.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.302+0000 7f6fb8ff9640 10 client.?.objecter ms_handle_connect 0x7f6fbc1457f0 2024-03-29T10:41:16.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.302+0000 7f6fb8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:16.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.302+0000 7f6fb8ff9640 1 -- 172.21.15.84:0/3220601363 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3953459148 0 0) 0x7f6fbc141f00 con 0x7f6fbc1457f0 2024-03-29T10:41:16.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.302+0000 7f6fb8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:41:16.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.302+0000 7f6fb8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:16.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.302+0000 7f6fb8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:16.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.302+0000 7f6fb8ff9640 10 monclient(hunting): my global_id is 4508 2024-03-29T10:41:16.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.302+0000 7f6fb8ff9640 1 -- 172.21.15.84:0/3220601363 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fa8004eb0 con 0x7f6fbc1457f0 2024-03-29T10:41:16.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.304+0000 7f6fb8ff9640 1 -- 172.21.15.84:0/3220601363 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2933207708 0 0) 0x7f6fbc140410 con 0x7f6fbc13f650 2024-03-29T10:41:16.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.304+0000 7f6fb8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:41:16.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.304+0000 7f6fb8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:16.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.304+0000 7f6fb8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:16.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.304+0000 7f6fb8ff9640 10 monclient(hunting): my global_id is 4449 2024-03-29T10:41:16.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.304+0000 7f6fb8ff9640 1 -- 172.21.15.84:0/3220601363 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fbc141f00 con 0x7f6fbc13f650 2024-03-29T10:41:16.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.304+0000 7f6fb8ff9640 1 -- 172.21.15.84:0/3220601363 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3024086309 0 0) 0x7f6fa8004eb0 con 0x7f6fbc1457f0 2024-03-29T10:41:16.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.304+0000 7f6fb8ff9640 1 -- 172.21.15.84:0/3220601363 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fbc140410 con 0x7f6fbc1457f0 2024-03-29T10:41:16.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.304+0000 7f6fb8ff9640 1 -- 172.21.15.84:0/3220601363 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f6fb4001950 con 0x7f6fbc1457f0 2024-03-29T10:41:16.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.304+0000 7f6fb8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:16.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.304+0000 7f6fb8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:16.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.304+0000 7f6fb8ff9640 10 monclient(hunting): dump: 2024-03-29T10:41:16.305 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:16.305 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:16.305 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:16.305 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:16.305 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:16.305 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:16.305 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:16.305 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:16.305 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:16.306 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:16.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.304+0000 7f6fb8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:16.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.304+0000 7f6fb8ff9640 10 client.?.objecter ms_handle_connect 0x7f6fbc0a9d70 2024-03-29T10:41:16.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.304+0000 7f6fb8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:16.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.305+0000 7f6fb8ff9640 1 -- 172.21.15.84:0/3220601363 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 919110977 0 0) 0x7f6fbc140410 con 0x7f6fbc1457f0 2024-03-29T10:41:16.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.305+0000 7f6fb8ff9640 1 -- 172.21.15.84:0/3220601363 >> v1:172.21.15.86:6789/0 conn(0x7f6fbc0a9d70 legacy=0x7f6fbc1417f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:16.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.305+0000 7f6fb8ff9640 1 -- 172.21.15.84:0/3220601363 >> v1:172.21.15.84:6789/0 conn(0x7f6fbc13f650 legacy=0x7f6fbc136970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:16.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.305+0000 7f6fb8ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:41:16.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.305+0000 7f6fb8ff9640 1 monclient: found mon.c 2024-03-29T10:41:16.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.305+0000 7f6fb8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:16.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.305+0000 7f6fb8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:41:16.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.305+0000 7f6fb8ff9640 1 -- 172.21.15.84:0/3220601363 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fbc137790 con 0x7f6fbc1457f0 2024-03-29T10:41:16.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.305+0000 7f6fb8ff9640 10 monclient: _finish_auth 0 2024-03-29T10:41:16.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.305+0000 7f6fb8ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:41:16.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.305+0000 7f6fb8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:16.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.305+0000 7f6fc24c0640 5 monclient: authenticate success, global_id 4508 2024-03-29T10:41:16.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.305+0000 7f6fc24c0640 10 monclient: _renew_subs 2024-03-29T10:41:16.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.305+0000 7f6fc24c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:41:16.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.305+0000 7f6fc24c0640 1 -- 172.21.15.84:0/3220601363 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fbc06ed90 con 0x7f6fbc1457f0 2024-03-29T10:41:16.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.305+0000 7f6fc24c0640 10 client.4508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:16.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.305+0000 7f6fc24c0640 10 monclient: _renew_subs 2024-03-29T10:41:16.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.305+0000 7f6fc24c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:41:16.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.305+0000 7f6fc24c0640 1 -- 172.21.15.84:0/3220601363 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fbc13fda0 con 0x7f6fbc1457f0 2024-03-29T10:41:16.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.305+0000 7f6fc24c0640 1 librados: init done 2024-03-29T10:41:16.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.306+0000 7f6fb8ff9640 1 -- 172.21.15.84:0/3220601363 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fb4003c20 con 0x7f6fbc1457f0 2024-03-29T10:41:16.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.306+0000 7f6fb8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:16.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.306+0000 7f6fb8ff9640 1 -- 172.21.15.84:0/3220601363 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f6fb4003f30 con 0x7f6fbc1457f0 2024-03-29T10:41:16.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.306+0000 7f6fba7fc640 4 set_mon_vals no callback set 2024-03-29T10:41:16.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.306+0000 7f6fb8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:16.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.307+0000 7f6fb8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:16.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.307+0000 7f6fb8ff9640 10 monclient: dump: 2024-03-29T10:41:16.308 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:16.308 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:16.308 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:16.308 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:16.308 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:16.308 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:16.308 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:16.308 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:16.309 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:16.309 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:16.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.307+0000 7f6fb8ff9640 1 -- 172.21.15.84:0/3220601363 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f6fb4019f00 con 0x7f6fbc1457f0 2024-03-29T10:41:16.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.307+0000 7f6fc24c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:16.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.307+0000 7f6fc24c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:16.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.307+0000 7f6fc24c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:41:16.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.307+0000 7f6fc24c0640 1 -- 172.21.15.84:0/3220601363 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f9c005080 con 0x7f6fbc1457f0 2024-03-29T10:41:16.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.308+0000 7f6fbbfff640 10 monclient: get_auth_request con 0x7f6fa806dd30 auth_method 0 2024-03-29T10:41:16.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.308+0000 7f6fb8ff9640 1 -- 172.21.15.84:0/3220601363 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f6fb4083c90 con 0x7f6fbc1457f0 2024-03-29T10:41:16.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.308+0000 7f6fb8ff9640 10 client.4508.objecter ms_dispatch 0x7f6fbc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:16.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.308+0000 7f6fb8ff9640 3 client.4508.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:16.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.309+0000 7f6fb8ff9640 3 client.4508.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:16.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.309+0000 7f6fb8ff9640 20 client.4508.objecter dump_active .. 0 homeless 2024-03-29T10:41:16.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.309+0000 7f6fb8ff9640 10 client.4508.objecter ms_handle_connect 0x7f6fa806dd30 2024-03-29T10:41:16.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.315+0000 7f6fb8ff9640 1 -- 172.21.15.84:0/3220601363 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6fb404f570 con 0x7f6fbc1457f0 2024-03-29T10:41:16.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.315+0000 7f6fb8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:16.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.315+0000 7f6fb8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:16.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.622+0000 7f6fc24c0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:16.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.622+0000 7f6fc24c0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:16.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.622+0000 7f6fc24c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:41:16.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.622+0000 7f6fc24c0640 1 -- 172.21.15.84:0/3220601363 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f9c005330 con 0x7f6fbc1457f0 2024-03-29T10:41:16.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.622+0000 7f6fb8ff9640 1 -- 172.21.15.84:0/3220601363 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f6fb40514a0 con 0x7f6fbc1457f0 2024-03-29T10:41:16.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.623+0000 7f6fb8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:16.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.623+0000 7f6fb8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:16.607 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:41:16.607 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:16.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.625+0000 7f6fc24c0640 10 librados: watch_flush enter 2024-03-29T10:41:16.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.626+0000 7f6fc24c0640 10 librados: watch_flush exit 2024-03-29T10:41:16.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.626+0000 7f6fc24c0640 20 client.4508.objecter shutdown clearing up homeless session... 2024-03-29T10:41:16.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.626+0000 7f6fc24c0640 10 client.4508.objecter successfully canceled tick 2024-03-29T10:41:16.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.626+0000 7f6fc24c0640 1 -- 172.21.15.84:0/3220601363 >> v1:172.21.15.86:6816/3337225337 conn(0x7f6fa806dd30 legacy=0x7f6fa80701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:16.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.626+0000 7f6fc24c0640 10 monclient: shutdown 2024-03-29T10:41:16.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.626+0000 7f6fc24c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:16.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.626+0000 7f6fc24c0640 1 -- 172.21.15.84:0/3220601363 >> v1:172.21.15.84:6790/0 conn(0x7f6fbc1457f0 legacy=0x7f6fbc137080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:16.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.626+0000 7f6fc24c0640 1 -- 172.21.15.84:0/3220601363 shutdown_connections 2024-03-29T10:41:16.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.626+0000 7f6fc24c0640 1 -- 172.21.15.84:0/3220601363 >> 172.21.15.84:0/3220601363 conn(0x7f6fbc066a00 msgr2=0x7f6fbc13b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:16.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.626+0000 7f6fc24c0640 1 -- 172.21.15.84:0/3220601363 shutdown_connections 2024-03-29T10:41:16.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.626+0000 7f6fc24c0640 1 -- 172.21.15.84:0/3220601363 wait complete. 2024-03-29T10:41:16.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:16.626+0000 7f6fc24c0640 1 librados: shutdown 2024-03-29T10:41:16.624 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:23.432 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:23.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.607+0000 7ff08c94d640 10 monclient: get_monmap_and_config 2024-03-29T10:41:23.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.607+0000 7ff08c94d640 10 monclient: build_initial_monmap 2024-03-29T10:41:23.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.607+0000 7ff08c94d640 10 monclient: monmap: 2024-03-29T10:41:23.591 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:41:23.591 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:23.591 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:41:23.608327+0000 2024-03-29T10:41:23.591 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:41:23.608327+0000 2024-03-29T10:41:23.591 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:23.591 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:23.591 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:41:23.591 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:41:23.591 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:41:23.591 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:23.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.608+0000 7ff08c94d640 1 Processor -- start 2024-03-29T10:41:23.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.608+0000 7ff08c94d640 1 -- start start 2024-03-29T10:41:23.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.608+0000 7ff08c94d640 10 monclient: init 2024-03-29T10:41:23.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.609+0000 7ff08c94d640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:23.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.609+0000 7ff08c94d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:41:23.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.609+0000 7ff08c94d640 10 monclient(hunting): picked mon.noname-c con 0x7ff0881422b0 addr v1:172.21.15.86:6789/0 2024-03-29T10:41:23.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.609+0000 7ff08c94d640 10 monclient(hunting): picked mon.noname-a con 0x7ff088142690 addr v1:172.21.15.84:6789/0 2024-03-29T10:41:23.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.609+0000 7ff08c94d640 10 monclient(hunting): picked mon.noname-b con 0x7ff0880b17d0 addr v1:172.21.15.84:6790/0 2024-03-29T10:41:23.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.609+0000 7ff08c94d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff088142c80 con 0x7ff088142690 2024-03-29T10:41:23.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.609+0000 7ff08c94d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0880aae40 con 0x7ff0881422b0 2024-03-29T10:41:23.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.609+0000 7ff08c94d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08813fbd0 con 0x7ff0880b17d0 2024-03-29T10:41:23.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.609+0000 7ff08c94d640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:23.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.609+0000 7ff08c94d640 10 monclient(hunting): authenticate will time out at 1185.872652s 2024-03-29T10:41:23.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.609+0000 7ff086d9d640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7ff0880b17d0 0x7ff0880b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:38278/0 (socket says 172.21.15.84:38278) 2024-03-29T10:41:23.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.609+0000 7ff086d9d640 1 -- 172.21.15.84:0/2012396703 learned_addr learned my addr 172.21.15.84:0/2012396703 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:41:23.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 1 -- 172.21.15.84:0/2012396703 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 692188222 0 0) 0x7ff08813fbd0 con 0x7ff0880b17d0 2024-03-29T10:41:23.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 10 monclient(hunting): _negotiate 2024-03-29T10:41:23.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:23.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:23.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 10 monclient(hunting): my global_id is 4511 2024-03-29T10:41:23.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 1 -- 172.21.15.84:0/2012396703 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff070003520 con 0x7ff0880b17d0 2024-03-29T10:41:23.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 1 -- 172.21.15.84:0/2012396703 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2875084635 0 0) 0x7ff0880aae40 con 0x7ff0881422b0 2024-03-29T10:41:23.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 10 monclient(hunting): _negotiate 2024-03-29T10:41:23.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:23.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:23.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 10 monclient(hunting): my global_id is 4420 2024-03-29T10:41:23.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 1 -- 172.21.15.84:0/2012396703 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff08813fbd0 con 0x7ff0881422b0 2024-03-29T10:41:23.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 1 -- 172.21.15.84:0/2012396703 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3883750845 0 0) 0x7ff088142c80 con 0x7ff088142690 2024-03-29T10:41:23.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 10 monclient(hunting): _negotiate 2024-03-29T10:41:23.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:23.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:23.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 10 monclient(hunting): my global_id is 4452 2024-03-29T10:41:23.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 1 -- 172.21.15.84:0/2012396703 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0880aae40 con 0x7ff088142690 2024-03-29T10:41:23.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 1 -- 172.21.15.84:0/2012396703 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1297974851 0 0) 0x7ff070003520 con 0x7ff0880b17d0 2024-03-29T10:41:23.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 1 -- 172.21.15.84:0/2012396703 >> v1:172.21.15.86:6789/0 conn(0x7ff0881422b0 legacy=0x7ff08813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:23.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 1 -- 172.21.15.84:0/2012396703 >> v1:172.21.15.84:6789/0 conn(0x7ff088142690 legacy=0x7ff0880b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:23.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 10 monclient: _finish_hunting 0 2024-03-29T10:41:23.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 1 monclient: found mon.noname-b 2024-03-29T10:41:23.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:23.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:41:23.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 1 -- 172.21.15.84:0/2012396703 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff088151940 con 0x7ff0880b17d0 2024-03-29T10:41:23.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 10 monclient: _finish_auth 0 2024-03-29T10:41:23.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 10 monclient: _check_auth_tickets 2024-03-29T10:41:23.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:23.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08c94d640 5 monclient: authenticate success, global_id 4511 2024-03-29T10:41:23.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08c94d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:23.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 1 -- 172.21.15.84:0/2012396703 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7ff080001a10 con 0x7ff0880b17d0 2024-03-29T10:41:23.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:23.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:23.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08559a640 10 monclient: dump: 2024-03-29T10:41:23.597 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:23.597 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:23.597 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:23.597 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:23.597 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:23.597 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:23.597 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:23.597 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:23.597 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:23.597 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:23.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.610+0000 7ff08c94d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:23.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.611+0000 7ff08559a640 1 -- 172.21.15.84:0/2012396703 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff080001c30 con 0x7ff0880b17d0 2024-03-29T10:41:23.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.611+0000 7ff08559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:23.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.611+0000 7ff08c94d640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:23.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.611+0000 7ff08c94d640 4 set_mon_vals no callback set 2024-03-29T10:41:23.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.611+0000 7ff08559a640 1 -- 172.21.15.84:0/2012396703 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7ff080001f40 con 0x7ff0880b17d0 2024-03-29T10:41:23.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.611+0000 7ff08559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:23.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.611+0000 7ff08559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:23.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.611+0000 7ff08559a640 10 monclient: dump: 2024-03-29T10:41:23.601 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:23.601 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:23.601 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:23.601 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:23.601 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:23.601 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:23.601 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:23.601 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:23.601 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:23.601 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:23.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.612+0000 7ff08c94d640 10 monclient: shutdown 2024-03-29T10:41:23.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.612+0000 7ff08c94d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:23.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.612+0000 7ff08c94d640 1 -- 172.21.15.84:0/2012396703 >> v1:172.21.15.84:6790/0 conn(0x7ff0880b17d0 legacy=0x7ff0880b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:23.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.612+0000 7ff08c94d640 1 -- 172.21.15.84:0/2012396703 shutdown_connections 2024-03-29T10:41:23.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.612+0000 7ff08c94d640 1 -- 172.21.15.84:0/2012396703 >> 172.21.15.84:0/2012396703 conn(0x7ff088066a00 msgr2=0x7ff088066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:23.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.612+0000 7ff08c94d640 1 -- 172.21.15.84:0/2012396703 shutdown_connections 2024-03-29T10:41:23.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.612+0000 7ff08c94d640 1 -- 172.21.15.84:0/2012396703 wait complete. 2024-03-29T10:41:23.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.613+0000 7ff08c94d640 10 monclient: build_initial_monmap 2024-03-29T10:41:23.602 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.613+0000 7ff08c94d640 10 monclient: monmap: 2024-03-29T10:41:23.602 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:41:23.602 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:23.602 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:41:23.602 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:41:23.602 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:23.602 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:23.603 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:41:23.603 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:41:23.603 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:41:23.603 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:23.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.613+0000 7ff08c94d640 1 librados: starting msgr at 2024-03-29T10:41:23.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.613+0000 7ff08c94d640 1 librados: starting objecter 2024-03-29T10:41:23.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.614+0000 7ff08c94d640 1 Processor -- start 2024-03-29T10:41:23.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.614+0000 7ff08c94d640 1 -- start start 2024-03-29T10:41:23.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.614+0000 7ff08c94d640 1 librados: setting wanted keys 2024-03-29T10:41:23.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.614+0000 7ff08c94d640 1 librados: calling monclient init 2024-03-29T10:41:23.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.614+0000 7ff08c94d640 10 monclient: init 2024-03-29T10:41:23.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.614+0000 7ff08c94d640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:23.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.614+0000 7ff08c94d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:41:23.603 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.614+0000 7ff08c94d640 10 monclient(hunting): picked mon.noname-b con 0x7ff0880b17d0 addr v1:172.21.15.86:6789/0 2024-03-29T10:41:23.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.615+0000 7ff08c94d640 10 monclient(hunting): picked mon.noname-c con 0x7ff0881422b0 addr v1:172.21.15.84:6790/0 2024-03-29T10:41:23.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.615+0000 7ff08659c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7ff0880b17d0 0x7ff08812fcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:59268/0 (socket says 172.21.15.84:59268) 2024-03-29T10:41:23.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.615+0000 7ff08659c640 1 -- 172.21.15.84:0/3947212748 learned_addr learned my addr 172.21.15.84:0/3947212748 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:41:23.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.615+0000 7ff076ffd640 10 client.?.objecter ms_handle_connect 0x7ff0880b17d0 2024-03-29T10:41:23.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.615+0000 7ff076ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:23.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.615+0000 7ff076ffd640 10 client.?.objecter ms_handle_connect 0x7ff0881422b0 2024-03-29T10:41:23.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.615+0000 7ff076ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:23.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.615+0000 7ff08c94d640 10 monclient(hunting): picked mon.noname-a con 0x7ff088142690 addr v1:172.21.15.84:6789/0 2024-03-29T10:41:23.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.615+0000 7ff076ffd640 10 client.?.objecter ms_handle_connect 0x7ff088142690 2024-03-29T10:41:23.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.615+0000 7ff076ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:23.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.615+0000 7ff08c94d640 1 -- 172.21.15.84:0/3947212748 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08812f680 con 0x7ff088142690 2024-03-29T10:41:23.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.615+0000 7ff08c94d640 1 -- 172.21.15.84:0/3947212748 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08806ecd0 con 0x7ff0880b17d0 2024-03-29T10:41:23.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.615+0000 7ff08c94d640 1 -- 172.21.15.84:0/3947212748 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08806efb0 con 0x7ff0881422b0 2024-03-29T10:41:23.604 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.615+0000 7ff08c94d640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:23.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.615+0000 7ff08c94d640 10 monclient(hunting): authenticate will time out at 1185.878639s 2024-03-29T10:41:23.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.615+0000 7ff076ffd640 1 -- 172.21.15.84:0/3947212748 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2921386596 0 0) 0x7ff08806ecd0 con 0x7ff0880b17d0 2024-03-29T10:41:23.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.615+0000 7ff076ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:23.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.615+0000 7ff076ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:23.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.615+0000 7ff076ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:23.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.615+0000 7ff076ffd640 10 monclient(hunting): my global_id is 4423 2024-03-29T10:41:23.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.615+0000 7ff076ffd640 1 -- 172.21.15.84:0/3947212748 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff068004eb0 con 0x7ff0880b17d0 2024-03-29T10:41:23.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.615+0000 7ff076ffd640 1 -- 172.21.15.84:0/3947212748 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890949391 0 0) 0x7ff08806efb0 con 0x7ff0881422b0 2024-03-29T10:41:23.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.615+0000 7ff076ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:23.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.615+0000 7ff076ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:23.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.615+0000 7ff076ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:23.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.615+0000 7ff076ffd640 10 monclient(hunting): my global_id is 4514 2024-03-29T10:41:23.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.615+0000 7ff076ffd640 1 -- 172.21.15.84:0/3947212748 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff08806ecd0 con 0x7ff0881422b0 2024-03-29T10:41:23.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 1 -- 172.21.15.84:0/3947212748 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2399232310 0 0) 0x7ff068004eb0 con 0x7ff0880b17d0 2024-03-29T10:41:23.605 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 1 -- 172.21.15.84:0/3947212748 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff08806efb0 con 0x7ff0880b17d0 2024-03-29T10:41:23.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 1 -- 172.21.15.84:0/3947212748 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7ff078001a40 con 0x7ff0880b17d0 2024-03-29T10:41:23.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:23.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:23.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 10 monclient(hunting): dump: 2024-03-29T10:41:23.606 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:23.606 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:23.606 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:23.606 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:23.606 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:23.606 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:23.606 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:23.606 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:23.606 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:23.606 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:23.606 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:23.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 1 -- 172.21.15.84:0/3947212748 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3923885527 0 0) 0x7ff08806ecd0 con 0x7ff0881422b0 2024-03-29T10:41:23.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 1 -- 172.21.15.84:0/3947212748 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff068004eb0 con 0x7ff0881422b0 2024-03-29T10:41:23.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 1 -- 172.21.15.84:0/3947212748 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3567788443 0 0) 0x7ff08812f680 con 0x7ff088142690 2024-03-29T10:41:23.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:23.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:23.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:23.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 10 monclient(hunting): my global_id is 4455 2024-03-29T10:41:23.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 1 -- 172.21.15.84:0/3947212748 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff08806ecd0 con 0x7ff088142690 2024-03-29T10:41:23.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 1 -- 172.21.15.84:0/3947212748 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7ff07c001890 con 0x7ff0881422b0 2024-03-29T10:41:23.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:23.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:23.607 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 10 monclient(hunting): dump: 2024-03-29T10:41:23.607 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:23.607 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:23.608 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:23.608 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:23.608 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:23.608 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:23.608 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:23.608 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:23.608 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:23.608 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:23.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 1 -- 172.21.15.84:0/3947212748 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1546670419 0 0) 0x7ff08806efb0 con 0x7ff0880b17d0 2024-03-29T10:41:23.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 1 -- 172.21.15.84:0/3947212748 >> v1:172.21.15.84:6790/0 conn(0x7ff0881422b0 legacy=0x7ff08812e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:23.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 1 -- 172.21.15.84:0/3947212748 >> v1:172.21.15.84:6789/0 conn(0x7ff088142690 legacy=0x7ff08812ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:23.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:41:23.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 1 monclient: found mon.b 2024-03-29T10:41:23.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:23.608 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:41:23.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 1 -- 172.21.15.84:0/3947212748 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0881303d0 con 0x7ff0880b17d0 2024-03-29T10:41:23.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 10 monclient: _finish_auth 0 2024-03-29T10:41:23.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:41:23.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff076ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:23.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff08c94d640 5 monclient: authenticate success, global_id 4423 2024-03-29T10:41:23.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff08c94d640 10 monclient: _renew_subs 2024-03-29T10:41:23.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff08c94d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:41:23.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff08c94d640 1 -- 172.21.15.84:0/3947212748 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff088131de0 con 0x7ff0880b17d0 2024-03-29T10:41:23.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff08c94d640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:23.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff08c94d640 10 monclient: _renew_subs 2024-03-29T10:41:23.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff08c94d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:41:23.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff08c94d640 1 -- 172.21.15.84:0/3947212748 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff088150ba0 con 0x7ff0880b17d0 2024-03-29T10:41:23.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.616+0000 7ff08c94d640 1 librados: init done 2024-03-29T10:41:23.609 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.617+0000 7ff076ffd640 1 -- 172.21.15.84:0/3947212748 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff078001c60 con 0x7ff0880b17d0 2024-03-29T10:41:23.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.617+0000 7ff076ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:23.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.617+0000 7ff076ffd640 1 -- 172.21.15.84:0/3947212748 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7ff078003a40 con 0x7ff0880b17d0 2024-03-29T10:41:23.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.617+0000 7ff076ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:23.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.617+0000 7ff076ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:23.610 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.617+0000 7ff076ffd640 10 monclient: dump: 2024-03-29T10:41:23.610 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:23.610 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:23.610 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:23.610 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:23.610 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:23.610 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:23.610 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:23.610 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:23.611 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:23.611 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:23.611 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.617+0000 7ff084d99640 4 set_mon_vals no callback set 2024-03-29T10:41:23.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.618+0000 7ff08c94d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:23.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.618+0000 7ff08c94d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:23.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.618+0000 7ff08c94d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:41:23.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.618+0000 7ff08c94d640 1 -- 172.21.15.84:0/3947212748 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff06c005080 con 0x7ff0880b17d0 2024-03-29T10:41:23.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.619+0000 7ff076ffd640 1 -- 172.21.15.84:0/3947212748 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7ff078003ba0 con 0x7ff0880b17d0 2024-03-29T10:41:23.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.619+0000 7ff076ffd640 1 -- 172.21.15.84:0/3947212748 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7ff078084a40 con 0x7ff0880b17d0 2024-03-29T10:41:23.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.619+0000 7ff076ffd640 10 client.4423.objecter ms_dispatch 0x7ff088000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:23.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.619+0000 7ff076ffd640 3 client.4423.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:23.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.619+0000 7ff076ffd640 3 client.4423.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:23.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.619+0000 7ff076ffd640 20 client.4423.objecter dump_active .. 0 homeless 2024-03-29T10:41:23.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.619+0000 7ff085d9b640 10 monclient: get_auth_request con 0x7ff06806df30 auth_method 0 2024-03-29T10:41:23.612 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.620+0000 7ff076ffd640 10 client.4423.objecter ms_handle_connect 0x7ff06806df30 2024-03-29T10:41:23.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.629+0000 7ff076ffd640 1 -- 172.21.15.84:0/3947212748 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff078050380 con 0x7ff0880b17d0 2024-03-29T10:41:23.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.629+0000 7ff076ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:23.613 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.629+0000 7ff076ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:23.966 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.982+0000 7ff08c94d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:23.966 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.983+0000 7ff08c94d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:23.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.983+0000 7ff08c94d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:41:23.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.983+0000 7ff08c94d640 1 -- 172.21.15.84:0/3947212748 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff06c005330 con 0x7ff0880b17d0 2024-03-29T10:41:23.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.984+0000 7ff076ffd640 1 -- 172.21.15.84:0/3947212748 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7ff0780840f0 con 0x7ff0880b17d0 2024-03-29T10:41:23.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.984+0000 7ff076ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:23.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.984+0000 7ff076ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:23.967 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:41:23.968 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:23.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.987+0000 7ff08c94d640 10 librados: watch_flush enter 2024-03-29T10:41:23.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.987+0000 7ff08c94d640 10 librados: watch_flush exit 2024-03-29T10:41:23.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.987+0000 7ff08c94d640 20 client.4423.objecter shutdown clearing up homeless session... 2024-03-29T10:41:23.972 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.987+0000 7ff08c94d640 10 client.4423.objecter successfully canceled tick 2024-03-29T10:41:23.972 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.987+0000 7ff08c94d640 1 -- 172.21.15.84:0/3947212748 >> v1:172.21.15.86:6816/3337225337 conn(0x7ff06806df30 legacy=0x7ff0680703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:23.972 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.987+0000 7ff08c94d640 10 monclient: shutdown 2024-03-29T10:41:23.972 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.987+0000 7ff08c94d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:23.972 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.987+0000 7ff08c94d640 1 -- 172.21.15.84:0/3947212748 >> v1:172.21.15.86:6789/0 conn(0x7ff0880b17d0 legacy=0x7ff08812fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:23.972 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.987+0000 7ff08c94d640 1 -- 172.21.15.84:0/3947212748 shutdown_connections 2024-03-29T10:41:23.972 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.987+0000 7ff08c94d640 1 -- 172.21.15.84:0/3947212748 >> 172.21.15.84:0/3947212748 conn(0x7ff088066a00 msgr2=0x7ff08813f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:23.972 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.988+0000 7ff08c94d640 1 -- 172.21.15.84:0/3947212748 shutdown_connections 2024-03-29T10:41:23.972 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.988+0000 7ff08c94d640 1 -- 172.21.15.84:0/3947212748 wait complete. 2024-03-29T10:41:23.972 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:23.988+0000 7ff08c94d640 1 librados: shutdown 2024-03-29T10:41:23.985 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:30.792 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:30.941 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.957+0000 7fb4232a8640 10 monclient: get_monmap_and_config 2024-03-29T10:41:30.941 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.957+0000 7fb4232a8640 10 monclient: build_initial_monmap 2024-03-29T10:41:30.941 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.957+0000 7fb4232a8640 10 monclient: monmap: 2024-03-29T10:41:30.941 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:41:30.941 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:30.942 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:41:30.958578+0000 2024-03-29T10:41:30.942 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:41:30.958578+0000 2024-03-29T10:41:30.942 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:30.942 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:30.942 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:41:30.942 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:41:30.942 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:41:30.942 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:30.944 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.960+0000 7fb4232a8640 1 Processor -- start 2024-03-29T10:41:30.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.961+0000 7fb4232a8640 1 -- start start 2024-03-29T10:41:30.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.961+0000 7fb4232a8640 10 monclient: init 2024-03-29T10:41:30.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.961+0000 7fb4232a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:30.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.961+0000 7fb4232a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:41:30.945 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.962+0000 7fb4232a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb41c0b4920 addr v1:172.21.15.86:6789/0 2024-03-29T10:41:30.946 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.962+0000 7fb4232a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb41c0b1790 addr v1:172.21.15.84:6789/0 2024-03-29T10:41:30.946 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.962+0000 7fb4232a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb41c146790 addr v1:172.21.15.84:6790/0 2024-03-29T10:41:30.946 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.962+0000 7fb4232a8640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb41c0b4070 con 0x7fb41c0b1790 2024-03-29T10:41:30.946 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.962+0000 7fb4232a8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb41c06fb80 con 0x7fb41c0b4920 2024-03-29T10:41:30.946 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.962+0000 7fb4232a8640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb41c0b42c0 con 0x7fb41c146790 2024-03-29T10:41:30.946 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.962+0000 7fb4232a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:30.946 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.962+0000 7fb4232a8640 10 monclient(hunting): authenticate will time out at 1193.225780s 2024-03-29T10:41:30.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.965+0000 7fb421044640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fb41c0b4920 0x7fb41c06e510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:37285/0 (socket says 172.21.15.84:37285) 2024-03-29T10:41:30.949 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.965+0000 7fb421044640 1 -- 172.21.15.84:0/1048181323 learned_addr learned my addr 172.21.15.84:0/1048181323 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:41:30.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.966+0000 7fb413fff640 1 -- 172.21.15.84:0/1048181323 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2039528891 0 0) 0x7fb41c06fb80 con 0x7fb41c0b4920 2024-03-29T10:41:30.950 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.966+0000 7fb413fff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:30.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.966+0000 7fb413fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:30.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.966+0000 7fb413fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:30.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.966+0000 7fb413fff640 10 monclient(hunting): my global_id is 4426 2024-03-29T10:41:30.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.966+0000 7fb413fff640 1 -- 172.21.15.84:0/1048181323 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb408003520 con 0x7fb41c0b4920 2024-03-29T10:41:30.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.966+0000 7fb413fff640 1 -- 172.21.15.84:0/1048181323 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3499603730 0 0) 0x7fb41c0b4070 con 0x7fb41c0b1790 2024-03-29T10:41:30.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.967+0000 7fb413fff640 10 monclient(hunting): _negotiate 2024-03-29T10:41:30.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.967+0000 7fb413fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:30.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.967+0000 7fb413fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:30.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.967+0000 7fb413fff640 10 monclient(hunting): my global_id is 4458 2024-03-29T10:41:30.951 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.967+0000 7fb413fff640 1 -- 172.21.15.84:0/1048181323 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb41c06fb80 con 0x7fb41c0b1790 2024-03-29T10:41:30.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.968+0000 7fb413fff640 1 -- 172.21.15.84:0/1048181323 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 948178426 0 0) 0x7fb408003520 con 0x7fb41c0b4920 2024-03-29T10:41:30.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.968+0000 7fb413fff640 1 -- 172.21.15.84:0/1048181323 >> v1:172.21.15.84:6790/0 conn(0x7fb41c146790 legacy=0x7fb41c146c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:41:30.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.968+0000 7fb413fff640 1 -- 172.21.15.84:0/1048181323 >> v1:172.21.15.84:6789/0 conn(0x7fb41c0b1790 legacy=0x7fb41c0b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:30.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.968+0000 7fb413fff640 10 monclient: _finish_hunting 0 2024-03-29T10:41:30.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.968+0000 7fb413fff640 1 monclient: found mon.noname-c 2024-03-29T10:41:30.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.968+0000 7fb413fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:30.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.968+0000 7fb413fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-29T10:41:30.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.968+0000 7fb413fff640 1 -- 172.21.15.84:0/1048181323 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb41c0b22c0 con 0x7fb41c0b4920 2024-03-29T10:41:30.954 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.968+0000 7fb413fff640 10 monclient: _finish_auth 0 2024-03-29T10:41:30.955 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.968+0000 7fb413fff640 10 monclient: _check_auth_tickets 2024-03-29T10:41:30.955 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.968+0000 7fb413fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:30.955 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.968+0000 7fb413fff640 1 -- 172.21.15.84:0/1048181323 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fb414001a10 con 0x7fb41c0b4920 2024-03-29T10:41:30.955 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.968+0000 7fb413fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:30.955 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.968+0000 7fb413fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:30.955 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.968+0000 7fb413fff640 10 monclient: dump: 2024-03-29T10:41:30.955 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:30.955 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:30.955 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:30.955 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:30.955 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:30.955 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:30.955 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:30.955 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:30.955 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:30.955 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:30.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.968+0000 7fb413fff640 1 -- 172.21.15.84:0/1048181323 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb414001c30 con 0x7fb41c0b4920 2024-03-29T10:41:30.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.968+0000 7fb413fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:30.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.968+0000 7fb413fff640 1 -- 172.21.15.84:0/1048181323 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fb414001fc0 con 0x7fb41c0b4920 2024-03-29T10:41:30.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.968+0000 7fb413fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:30.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.968+0000 7fb413fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:30.956 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.968+0000 7fb413fff640 10 monclient: dump: 2024-03-29T10:41:30.956 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:30.956 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:30.956 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:30.956 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:30.957 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:30.957 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:30.957 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:30.957 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:30.957 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:30.957 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:30.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.969+0000 7fb4232a8640 5 monclient: authenticate success, global_id 4426 2024-03-29T10:41:30.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.969+0000 7fb4232a8640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:30.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.969+0000 7fb4232a8640 4 set_mon_vals no callback set 2024-03-29T10:41:30.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.971+0000 7fb4232a8640 10 monclient: shutdown 2024-03-29T10:41:30.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.971+0000 7fb4232a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:30.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.971+0000 7fb4232a8640 1 -- 172.21.15.84:0/1048181323 >> v1:172.21.15.86:6789/0 conn(0x7fb41c0b4920 legacy=0x7fb41c06e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:30.958 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.972+0000 7fb4232a8640 1 -- 172.21.15.84:0/1048181323 shutdown_connections 2024-03-29T10:41:30.958 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.972+0000 7fb4232a8640 1 -- 172.21.15.84:0/1048181323 >> 172.21.15.84:0/1048181323 conn(0x7fb41c066a00 msgr2=0x7fb41c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:30.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.973+0000 7fb4232a8640 1 -- 172.21.15.84:0/1048181323 shutdown_connections 2024-03-29T10:41:30.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.973+0000 7fb4232a8640 1 -- 172.21.15.84:0/1048181323 wait complete. 2024-03-29T10:41:30.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.974+0000 7fb4232a8640 10 monclient: build_initial_monmap 2024-03-29T10:41:30.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.974+0000 7fb4232a8640 10 monclient: monmap: 2024-03-29T10:41:30.959 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:41:30.959 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:30.959 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:41:30.959 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:41:30.959 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:30.959 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:30.959 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:41:30.959 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:41:30.959 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:41:30.959 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:30.960 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.975+0000 7fb4232a8640 1 librados: starting msgr at 2024-03-29T10:41:30.960 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.975+0000 7fb4232a8640 1 librados: starting objecter 2024-03-29T10:41:30.960 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.975+0000 7fb4232a8640 1 Processor -- start 2024-03-29T10:41:30.961 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.975+0000 7fb4232a8640 1 -- start start 2024-03-29T10:41:30.961 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.975+0000 7fb4232a8640 1 librados: setting wanted keys 2024-03-29T10:41:30.961 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.975+0000 7fb4232a8640 1 librados: calling monclient init 2024-03-29T10:41:30.961 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.975+0000 7fb4232a8640 10 monclient: init 2024-03-29T10:41:30.961 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.976+0000 7fb4232a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:30.961 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.976+0000 7fb4232a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:41:30.961 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.976+0000 7fb4232a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb41c0b1790 addr v1:172.21.15.84:6789/0 2024-03-29T10:41:30.961 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.976+0000 7fb421044640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fb41c0b1790 0x7fb41c1df170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:48588/0 (socket says 172.21.15.84:48588) 2024-03-29T10:41:30.961 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.976+0000 7fb421044640 1 -- 172.21.15.84:0/3663084594 learned_addr learned my addr 172.21.15.84:0/3663084594 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:41:30.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.976+0000 7fb4232a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb41c0b4920 addr v1:172.21.15.86:6789/0 2024-03-29T10:41:30.968 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.976+0000 7fb4232a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb41c146790 addr v1:172.21.15.84:6790/0 2024-03-29T10:41:30.968 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.978+0000 7fb411ffb640 10 client.?.objecter ms_handle_connect 0x7fb41c0b1790 2024-03-29T10:41:30.968 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.978+0000 7fb411ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:30.968 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.980+0000 7fb4232a8640 1 -- 172.21.15.84:0/3663084594 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb41c1dece0 con 0x7fb41c0b1790 2024-03-29T10:41:30.968 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.980+0000 7fb4232a8640 1 -- 172.21.15.84:0/3663084594 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb41c134760 con 0x7fb41c0b4920 2024-03-29T10:41:30.968 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.980+0000 7fb4232a8640 1 -- 172.21.15.84:0/3663084594 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb41c1df880 con 0x7fb41c146790 2024-03-29T10:41:30.968 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.980+0000 7fb4232a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:30.968 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.980+0000 7fb4232a8640 10 monclient(hunting): authenticate will time out at 1193.243279s 2024-03-29T10:41:30.968 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.980+0000 7fb411ffb640 1 -- 172.21.15.84:0/3663084594 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 191775680 0 0) 0x7fb41c1dece0 con 0x7fb41c0b1790 2024-03-29T10:41:30.968 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.980+0000 7fb411ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:41:30.968 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.980+0000 7fb411ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:30.968 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.980+0000 7fb411ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:30.968 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.980+0000 7fb411ffb640 10 monclient(hunting): my global_id is 4461 2024-03-29T10:41:30.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.980+0000 7fb411ffb640 1 -- 172.21.15.84:0/3663084594 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb400004eb0 con 0x7fb41c0b1790 2024-03-29T10:41:30.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.981+0000 7fb411ffb640 1 -- 172.21.15.84:0/3663084594 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 290113279 0 0) 0x7fb400004eb0 con 0x7fb41c0b1790 2024-03-29T10:41:30.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.981+0000 7fb411ffb640 1 -- 172.21.15.84:0/3663084594 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb41c1dece0 con 0x7fb41c0b1790 2024-03-29T10:41:30.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.981+0000 7fb411ffb640 1 -- 172.21.15.84:0/3663084594 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fb414001a10 con 0x7fb41c0b1790 2024-03-29T10:41:30.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.981+0000 7fb411ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:30.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.981+0000 7fb411ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:41:30.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.981+0000 7fb411ffb640 10 monclient(hunting): dump: 2024-03-29T10:41:30.969 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:30.969 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:30.969 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:30.969 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:30.969 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:30.969 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:30.969 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:30.970 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:30.970 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:30.970 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:30.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.981+0000 7fb411ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:30.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.981+0000 7fb411ffb640 1 -- 172.21.15.84:0/3663084594 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 730701015 0 0) 0x7fb41c1dece0 con 0x7fb41c0b1790 2024-03-29T10:41:30.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.981+0000 7fb411ffb640 1 -- 172.21.15.84:0/3663084594 >> v1:172.21.15.84:6790/0 conn(0x7fb41c146790 legacy=0x7fb41c132d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:30.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.981+0000 7fb411ffb640 1 -- 172.21.15.84:0/3663084594 >> v1:172.21.15.86:6789/0 conn(0x7fb41c0b4920 legacy=0x7fb41c1d86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:30.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.981+0000 7fb411ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:41:30.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.981+0000 7fb411ffb640 1 monclient: found mon.a 2024-03-29T10:41:30.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.981+0000 7fb411ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:30.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.981+0000 7fb411ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:41:30.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.981+0000 7fb411ffb640 1 -- 172.21.15.84:0/3663084594 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb41c1dfa10 con 0x7fb41c0b1790 2024-03-29T10:41:30.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.982+0000 7fb411ffb640 10 monclient: _finish_auth 0 2024-03-29T10:41:30.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.982+0000 7fb411ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:41:30.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.982+0000 7fb411ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:30.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.982+0000 7fb4232a8640 5 monclient: authenticate success, global_id 4461 2024-03-29T10:41:30.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.982+0000 7fb4232a8640 10 monclient: _renew_subs 2024-03-29T10:41:30.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.982+0000 7fb4232a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:41:30.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.982+0000 7fb4232a8640 1 -- 172.21.15.84:0/3663084594 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb41c1d8040 con 0x7fb41c0b1790 2024-03-29T10:41:30.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.982+0000 7fb4232a8640 10 client.4461.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:30.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.982+0000 7fb4232a8640 10 monclient: _renew_subs 2024-03-29T10:41:30.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.982+0000 7fb4232a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:41:30.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.982+0000 7fb4232a8640 1 -- 172.21.15.84:0/3663084594 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb41c1d8fb0 con 0x7fb41c0b1790 2024-03-29T10:41:30.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.982+0000 7fb4232a8640 1 librados: init done 2024-03-29T10:41:30.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.983+0000 7fb411ffb640 1 -- 172.21.15.84:0/3663084594 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb414004e10 con 0x7fb41c0b1790 2024-03-29T10:41:30.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.983+0000 7fb411ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:30.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.983+0000 7fb411ffb640 1 -- 172.21.15.84:0/3663084594 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fb414005260 con 0x7fb41c0b1790 2024-03-29T10:41:30.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.983+0000 7fb411ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:30.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.983+0000 7fb411ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:41:30.972 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.983+0000 7fb411ffb640 10 monclient: dump: 2024-03-29T10:41:30.972 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:30.972 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:30.972 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:30.972 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:30.972 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:30.972 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:30.972 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:30.972 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:30.972 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:30.972 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:30.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.985+0000 7fb411ffb640 1 -- 172.21.15.84:0/3663084594 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7fb4140053c0 con 0x7fb41c0b1790 2024-03-29T10:41:30.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.985+0000 7fb411ffb640 1 -- 172.21.15.84:0/3663084594 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7fb414084fc0 con 0x7fb41c0b1790 2024-03-29T10:41:30.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.985+0000 7fb411ffb640 10 client.4461.objecter ms_dispatch 0x7fb41c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:30.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.985+0000 7fb411ffb640 3 client.4461.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:30.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.985+0000 7fb411ffb640 3 client.4461.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:30.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.985+0000 7fb412ffd640 4 set_mon_vals no callback set 2024-03-29T10:41:30.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.986+0000 7fb411ffb640 20 client.4461.objecter dump_active .. 0 homeless 2024-03-29T10:41:30.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.986+0000 7fb4077fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:30.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.987+0000 7fb4077fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:30.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.987+0000 7fb4077fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:41:30.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.987+0000 7fb4077fe640 1 -- 172.21.15.84:0/3663084594 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3fc005080 con 0x7fb41c0b1790 2024-03-29T10:41:30.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.988+0000 7fb420843640 10 monclient: get_auth_request con 0x7fb40006db30 auth_method 0 2024-03-29T10:41:30.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.989+0000 7fb411ffb640 10 client.4461.objecter ms_handle_connect 0x7fb40006db30 2024-03-29T10:41:30.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.997+0000 7fb411ffb640 1 -- 172.21.15.84:0/3663084594 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb4140508a0 con 0x7fb41c0b1790 2024-03-29T10:41:30.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.997+0000 7fb411ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:30.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:30.997+0000 7fb411ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:31.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:31.324+0000 7fb4077fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:31.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:31.324+0000 7fb4077fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:31.307 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:31.324+0000 7fb4077fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:41:31.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:31.324+0000 7fb4077fe640 1 -- 172.21.15.84:0/3663084594 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3fc005330 con 0x7fb41c0b1790 2024-03-29T10:41:31.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:31.325+0000 7fb411ffb640 1 -- 172.21.15.84:0/3663084594 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7fb4140527d0 con 0x7fb41c0b1790 2024-03-29T10:41:31.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:31.325+0000 7fb411ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:31.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:31.325+0000 7fb411ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:31.308 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:41:31.309 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:31.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:31.328+0000 7fb4077fe640 10 librados: watch_flush enter 2024-03-29T10:41:31.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:31.328+0000 7fb4077fe640 10 librados: watch_flush exit 2024-03-29T10:41:31.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:31.328+0000 7fb4077fe640 20 client.4461.objecter shutdown clearing up homeless session... 2024-03-29T10:41:31.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:31.328+0000 7fb4077fe640 10 client.4461.objecter successfully canceled tick 2024-03-29T10:41:31.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:31.329+0000 7fb4077fe640 1 -- 172.21.15.84:0/3663084594 >> v1:172.21.15.86:6816/3337225337 conn(0x7fb40006db30 legacy=0x7fb40006fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:31.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:31.329+0000 7fb4077fe640 10 monclient: shutdown 2024-03-29T10:41:31.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:31.329+0000 7fb4077fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:31.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:31.329+0000 7fb4077fe640 1 -- 172.21.15.84:0/3663084594 >> v1:172.21.15.84:6789/0 conn(0x7fb41c0b1790 legacy=0x7fb41c1df170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:31.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:31.330+0000 7fb4077fe640 1 -- 172.21.15.84:0/3663084594 shutdown_connections 2024-03-29T10:41:31.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:31.330+0000 7fb4077fe640 1 -- 172.21.15.84:0/3663084594 >> 172.21.15.84:0/3663084594 conn(0x7fb41c066a00 msgr2=0x7fb41c146be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:31.314 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:31.331+0000 7fb4077fe640 1 -- 172.21.15.84:0/3663084594 shutdown_connections 2024-03-29T10:41:31.314 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:31.331+0000 7fb4077fe640 1 -- 172.21.15.84:0/3663084594 wait complete. 2024-03-29T10:41:31.314 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:31.331+0000 7fb4077fe640 1 librados: shutdown 2024-03-29T10:41:31.333 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:38.140 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:38.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.303+0000 7fde6e96d640 10 monclient: get_monmap_and_config 2024-03-29T10:41:38.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.303+0000 7fde6e96d640 10 monclient: build_initial_monmap 2024-03-29T10:41:38.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.303+0000 7fde6e96d640 10 monclient: monmap: 2024-03-29T10:41:38.287 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:41:38.287 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:38.287 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:41:38.304029+0000 2024-03-29T10:41:38.287 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:41:38.304029+0000 2024-03-29T10:41:38.287 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:38.287 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:38.287 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:41:38.287 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:41:38.287 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:41:38.287 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:38.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.304+0000 7fde6e96d640 1 Processor -- start 2024-03-29T10:41:38.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.304+0000 7fde6e96d640 1 -- start start 2024-03-29T10:41:38.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.304+0000 7fde6e96d640 10 monclient: init 2024-03-29T10:41:38.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.305+0000 7fde6e96d640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:38.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.305+0000 7fde6e96d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:41:38.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.306+0000 7fde6e96d640 10 monclient(hunting): picked mon.noname-c con 0x7fde680b4330 addr v1:172.21.15.86:6789/0 2024-03-29T10:41:38.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.306+0000 7fde6e96d640 10 monclient(hunting): picked mon.noname-a con 0x7fde681470b0 addr v1:172.21.15.84:6789/0 2024-03-29T10:41:38.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.307+0000 7fde6e96d640 10 monclient(hunting): picked mon.noname-b con 0x7fde681439b0 addr v1:172.21.15.84:6790/0 2024-03-29T10:41:38.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.307+0000 7fde6e96d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde6806f2a0 con 0x7fde681470b0 2024-03-29T10:41:38.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.307+0000 7fde6cf0a640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fde681439b0 0x7fde68151d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:34246/0 (socket says 172.21.15.84:34246) 2024-03-29T10:41:38.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.307+0000 7fde6cf0a640 1 -- 172.21.15.84:0/3650734171 learned_addr learned my addr 172.21.15.84:0/3650734171 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:41:38.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.307+0000 7fde6e96d640 1 -- 172.21.15.84:0/3650734171 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde68146b90 con 0x7fde680b4330 2024-03-29T10:41:38.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.307+0000 7fde6e96d640 1 -- 172.21.15.84:0/3650734171 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde680b2ca0 con 0x7fde681439b0 2024-03-29T10:41:38.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.307+0000 7fde6e96d640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:38.291 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.307+0000 7fde6e96d640 10 monclient(hunting): authenticate will time out at 1200.570778s 2024-03-29T10:41:38.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.308+0000 7fde66ffd640 1 -- 172.21.15.84:0/3650734171 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164985951 0 0) 0x7fde680b2ca0 con 0x7fde681439b0 2024-03-29T10:41:38.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.308+0000 7fde66ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:38.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.308+0000 7fde66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:38.292 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.308+0000 7fde66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:38.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.308+0000 7fde66ffd640 10 monclient(hunting): my global_id is 4517 2024-03-29T10:41:38.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.308+0000 7fde66ffd640 1 -- 172.21.15.84:0/3650734171 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde50003520 con 0x7fde681439b0 2024-03-29T10:41:38.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.308+0000 7fde66ffd640 1 -- 172.21.15.84:0/3650734171 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 502710843 0 0) 0x7fde50003520 con 0x7fde681439b0 2024-03-29T10:41:38.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.308+0000 7fde66ffd640 1 -- 172.21.15.84:0/3650734171 >> v1:172.21.15.86:6789/0 conn(0x7fde680b4330 legacy=0x7fde680b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:41:38.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.308+0000 7fde66ffd640 1 -- 172.21.15.84:0/3650734171 >> v1:172.21.15.84:6789/0 conn(0x7fde681470b0 legacy=0x7fde680b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:41:38.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.310+0000 7fde66ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:41:38.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.310+0000 7fde66ffd640 1 monclient: found mon.noname-b 2024-03-29T10:41:38.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.310+0000 7fde66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:38.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.310+0000 7fde66ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:41:38.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.311+0000 7fde66ffd640 1 -- 172.21.15.84:0/3650734171 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde6806e0c0 con 0x7fde681439b0 2024-03-29T10:41:38.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.311+0000 7fde66ffd640 10 monclient: _finish_auth 0 2024-03-29T10:41:38.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.311+0000 7fde66ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:41:38.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.311+0000 7fde66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:38.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.311+0000 7fde66ffd640 1 -- 172.21.15.84:0/3650734171 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fde60001a10 con 0x7fde681439b0 2024-03-29T10:41:38.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.311+0000 7fde66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:38.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.311+0000 7fde66ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:38.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.311+0000 7fde66ffd640 10 monclient: dump: 2024-03-29T10:41:38.296 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:38.296 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:38.296 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:38.296 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:38.296 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:38.296 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:38.296 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:38.297 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:38.297 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:38.297 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:38.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.311+0000 7fde6e96d640 5 monclient: authenticate success, global_id 4517 2024-03-29T10:41:38.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.311+0000 7fde6e96d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:38.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.314+0000 7fde66ffd640 1 -- 172.21.15.84:0/3650734171 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde60001ca0 con 0x7fde681439b0 2024-03-29T10:41:38.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.314+0000 7fde66ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:38.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.314+0000 7fde66ffd640 1 -- 172.21.15.84:0/3650734171 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fde60002030 con 0x7fde681439b0 2024-03-29T10:41:38.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.314+0000 7fde66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:38.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.314+0000 7fde66ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:38.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.314+0000 7fde66ffd640 10 monclient: dump: 2024-03-29T10:41:38.298 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:38.299 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:38.299 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:38.299 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:38.299 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:38.299 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:38.299 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:38.299 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:38.299 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:38.299 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:38.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.314+0000 7fde6e96d640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:38.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.314+0000 7fde6e96d640 4 set_mon_vals no callback set 2024-03-29T10:41:38.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.315+0000 7fde6e96d640 10 monclient: shutdown 2024-03-29T10:41:38.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.315+0000 7fde6e96d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:38.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.315+0000 7fde6e96d640 1 -- 172.21.15.84:0/3650734171 >> v1:172.21.15.84:6790/0 conn(0x7fde681439b0 legacy=0x7fde68151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:38.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.316+0000 7fde6e96d640 1 -- 172.21.15.84:0/3650734171 shutdown_connections 2024-03-29T10:41:38.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.316+0000 7fde6e96d640 1 -- 172.21.15.84:0/3650734171 >> 172.21.15.84:0/3650734171 conn(0x7fde68066a00 msgr2=0x7fde68066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:38.302 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.319+0000 7fde6e96d640 1 -- 172.21.15.84:0/3650734171 shutdown_connections 2024-03-29T10:41:38.302 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.319+0000 7fde6e96d640 1 -- 172.21.15.84:0/3650734171 wait complete. 2024-03-29T10:41:38.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.320+0000 7fde6e96d640 10 monclient: build_initial_monmap 2024-03-29T10:41:38.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.320+0000 7fde6e96d640 10 monclient: monmap: 2024-03-29T10:41:38.303 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:41:38.304 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:38.304 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:41:38.304 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:41:38.304 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:38.304 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:38.304 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:41:38.304 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:41:38.304 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:41:38.304 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:38.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.321+0000 7fde6e96d640 1 librados: starting msgr at 2024-03-29T10:41:38.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.321+0000 7fde6e96d640 1 librados: starting objecter 2024-03-29T10:41:38.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.321+0000 7fde6e96d640 1 Processor -- start 2024-03-29T10:41:38.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.321+0000 7fde6e96d640 1 -- start start 2024-03-29T10:41:38.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.321+0000 7fde6e96d640 1 librados: setting wanted keys 2024-03-29T10:41:38.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.321+0000 7fde6e96d640 1 librados: calling monclient init 2024-03-29T10:41:38.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.321+0000 7fde6e96d640 10 monclient: init 2024-03-29T10:41:38.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.322+0000 7fde6e96d640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:38.306 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.322+0000 7fde6e96d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:41:38.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.322+0000 7fde6e96d640 10 monclient(hunting): picked mon.noname-a con 0x7fde680b4330 addr v1:172.21.15.84:6789/0 2024-03-29T10:41:38.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.325+0000 7fde6e96d640 10 monclient(hunting): picked mon.noname-c con 0x7fde681439b0 addr v1:172.21.15.84:6790/0 2024-03-29T10:41:38.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.325+0000 7fde677fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fde681439b0 0x7fde681db930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:36720/0 (socket says 172.21.15.84:36720) 2024-03-29T10:41:38.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.325+0000 7fde677fe640 1 -- 172.21.15.84:0/2657801518 learned_addr learned my addr 172.21.15.84:0/2657801518 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:41:38.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.327+0000 7fde64ff9640 10 client.?.objecter ms_handle_connect 0x7fde681439b0 2024-03-29T10:41:38.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.327+0000 7fde64ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:38.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.327+0000 7fde64ff9640 10 client.?.objecter ms_handle_connect 0x7fde680b4330 2024-03-29T10:41:38.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.327+0000 7fde64ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:38.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.327+0000 7fde6e96d640 10 monclient(hunting): picked mon.noname-b con 0x7fde681470b0 addr v1:172.21.15.86:6789/0 2024-03-29T10:41:38.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.327+0000 7fde6e96d640 1 -- 172.21.15.84:0/2657801518 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde68146610 con 0x7fde680b4330 2024-03-29T10:41:38.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.327+0000 7fde6e96d640 1 -- 172.21.15.84:0/2657801518 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde68140cb0 con 0x7fde681470b0 2024-03-29T10:41:38.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.327+0000 7fde6e96d640 1 -- 172.21.15.84:0/2657801518 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde68140e40 con 0x7fde681439b0 2024-03-29T10:41:38.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.327+0000 7fde6e96d640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:38.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.327+0000 7fde6e96d640 10 monclient(hunting): authenticate will time out at 1200.590904s 2024-03-29T10:41:38.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.328+0000 7fde64ff9640 1 -- 172.21.15.84:0/2657801518 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2507690703 0 0) 0x7fde68146610 con 0x7fde680b4330 2024-03-29T10:41:38.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.328+0000 7fde64ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:41:38.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.328+0000 7fde64ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:38.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.328+0000 7fde64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:38.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.328+0000 7fde64ff9640 10 monclient(hunting): my global_id is 4464 2024-03-29T10:41:38.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.328+0000 7fde64ff9640 1 -- 172.21.15.84:0/2657801518 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde54004eb0 con 0x7fde680b4330 2024-03-29T10:41:38.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.331+0000 7fde64ff9640 10 client.?.objecter ms_handle_connect 0x7fde681470b0 2024-03-29T10:41:38.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.331+0000 7fde64ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:38.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.331+0000 7fde64ff9640 1 -- 172.21.15.84:0/2657801518 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2536714930 0 0) 0x7fde68140e40 con 0x7fde681439b0 2024-03-29T10:41:38.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.331+0000 7fde64ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:41:38.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.331+0000 7fde64ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:38.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.331+0000 7fde64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:38.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.331+0000 7fde64ff9640 10 monclient(hunting): my global_id is 4520 2024-03-29T10:41:38.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.332+0000 7fde64ff9640 1 -- 172.21.15.84:0/2657801518 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde68146610 con 0x7fde681439b0 2024-03-29T10:41:38.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.332+0000 7fde64ff9640 1 -- 172.21.15.84:0/2657801518 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1283536355 0 0) 0x7fde54004eb0 con 0x7fde680b4330 2024-03-29T10:41:38.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.332+0000 7fde64ff9640 1 -- 172.21.15.84:0/2657801518 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde68140e40 con 0x7fde680b4330 2024-03-29T10:41:38.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.332+0000 7fde64ff9640 1 -- 172.21.15.84:0/2657801518 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fde58001680 con 0x7fde680b4330 2024-03-29T10:41:38.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.332+0000 7fde64ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:38.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.332+0000 7fde64ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:41:38.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.332+0000 7fde64ff9640 10 monclient(hunting): dump: 2024-03-29T10:41:38.317 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:38.317 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:38.317 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:38.317 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:38.317 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:38.317 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:38.317 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:38.317 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:38.317 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:38.317 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:38.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.332+0000 7fde64ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:38.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.332+0000 7fde64ff9640 1 -- 172.21.15.84:0/2657801518 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2178932044 0 0) 0x7fde68140cb0 con 0x7fde681470b0 2024-03-29T10:41:38.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.332+0000 7fde64ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:41:38.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.332+0000 7fde64ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:38.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.332+0000 7fde64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:38.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.332+0000 7fde64ff9640 10 monclient(hunting): my global_id is 4429 2024-03-29T10:41:38.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.332+0000 7fde64ff9640 1 -- 172.21.15.84:0/2657801518 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde54004eb0 con 0x7fde681470b0 2024-03-29T10:41:38.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.333+0000 7fde64ff9640 1 -- 172.21.15.84:0/2657801518 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3741254172 0 0) 0x7fde54004eb0 con 0x7fde681470b0 2024-03-29T10:41:38.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.333+0000 7fde64ff9640 1 -- 172.21.15.84:0/2657801518 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde68140cb0 con 0x7fde681470b0 2024-03-29T10:41:38.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.333+0000 7fde64ff9640 1 -- 172.21.15.84:0/2657801518 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fde600018d0 con 0x7fde681470b0 2024-03-29T10:41:38.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.333+0000 7fde64ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:38.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.333+0000 7fde64ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:38.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.333+0000 7fde64ff9640 10 monclient(hunting): dump: 2024-03-29T10:41:38.320 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:38.320 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:38.320 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:38.320 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:38.320 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:38.321 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:38.321 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:38.321 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:38.321 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:38.321 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:38.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.334+0000 7fde64ff9640 1 -- 172.21.15.84:0/2657801518 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1830999239 0 0) 0x7fde68146610 con 0x7fde681439b0 2024-03-29T10:41:38.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.334+0000 7fde64ff9640 1 -- 172.21.15.84:0/2657801518 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde54004eb0 con 0x7fde681439b0 2024-03-29T10:41:38.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.334+0000 7fde64ff9640 1 -- 172.21.15.84:0/2657801518 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fde5c0018e0 con 0x7fde681439b0 2024-03-29T10:41:38.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.334+0000 7fde64ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:38.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.334+0000 7fde64ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:38.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.334+0000 7fde64ff9640 10 monclient(hunting): dump: 2024-03-29T10:41:38.321 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:38.321 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:38.322 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:38.322 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:38.322 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:38.322 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:38.322 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:38.322 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:38.322 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:38.322 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:38.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.334+0000 7fde64ff9640 1 -- 172.21.15.84:0/2657801518 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 304874866 0 0) 0x7fde68140cb0 con 0x7fde681470b0 2024-03-29T10:41:38.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.334+0000 7fde64ff9640 1 -- 172.21.15.84:0/2657801518 >> v1:172.21.15.84:6790/0 conn(0x7fde681439b0 legacy=0x7fde681db930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:38.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.334+0000 7fde64ff9640 1 -- 172.21.15.84:0/2657801518 >> v1:172.21.15.84:6789/0 conn(0x7fde680b4330 legacy=0x7fde681405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:38.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.334+0000 7fde64ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:41:38.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.334+0000 7fde64ff9640 1 monclient: found mon.b 2024-03-29T10:41:38.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.334+0000 7fde64ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:38.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.334+0000 7fde64ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:41:38.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.334+0000 7fde64ff9640 1 -- 172.21.15.84:0/2657801518 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde68140fd0 con 0x7fde681470b0 2024-03-29T10:41:38.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.334+0000 7fde64ff9640 10 monclient: _finish_auth 0 2024-03-29T10:41:38.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.334+0000 7fde64ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:41:38.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.334+0000 7fde64ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:38.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.334+0000 7fde6e96d640 5 monclient: authenticate success, global_id 4429 2024-03-29T10:41:38.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.334+0000 7fde6e96d640 10 monclient: _renew_subs 2024-03-29T10:41:38.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.334+0000 7fde6e96d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:41:38.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.334+0000 7fde6e96d640 1 -- 172.21.15.84:0/2657801518 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde6811fda0 con 0x7fde681470b0 2024-03-29T10:41:38.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.335+0000 7fde6e96d640 10 client.4429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:38.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.335+0000 7fde6e96d640 10 monclient: _renew_subs 2024-03-29T10:41:38.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.335+0000 7fde6e96d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:41:38.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.335+0000 7fde6e96d640 1 -- 172.21.15.84:0/2657801518 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fde681202d0 con 0x7fde681470b0 2024-03-29T10:41:38.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.335+0000 7fde6e96d640 1 librados: init done 2024-03-29T10:41:38.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.335+0000 7fde64ff9640 1 -- 172.21.15.84:0/2657801518 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde60001a30 con 0x7fde681470b0 2024-03-29T10:41:38.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.335+0000 7fde64ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:38.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.335+0000 7fde64ff9640 1 -- 172.21.15.84:0/2657801518 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fde60005110 con 0x7fde681470b0 2024-03-29T10:41:38.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.335+0000 7fde64ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:38.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.335+0000 7fde64ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:38.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.335+0000 7fde64ff9640 10 monclient: dump: 2024-03-29T10:41:38.324 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:38.325 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:38.325 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:38.325 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:38.325 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:38.325 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:38.325 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:38.325 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:38.325 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:38.325 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:38.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.337+0000 7fde64ff9640 1 -- 172.21.15.84:0/2657801518 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7fde6001b0e0 con 0x7fde681470b0 2024-03-29T10:41:38.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.337+0000 7fde667fc640 4 set_mon_vals no callback set 2024-03-29T10:41:38.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.337+0000 7fde6e96d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:38.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.338+0000 7fde64ff9640 1 -- 172.21.15.84:0/2657801518 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7fde60084d50 con 0x7fde681470b0 2024-03-29T10:41:38.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.338+0000 7fde64ff9640 10 client.4429.objecter ms_dispatch 0x7fde68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:38.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.338+0000 7fde6e96d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:38.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.338+0000 7fde6e96d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:41:38.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.338+0000 7fde64ff9640 3 client.4429.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:38.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.338+0000 7fde64ff9640 3 client.4429.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:38.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.338+0000 7fde64ff9640 20 client.4429.objecter dump_active .. 0 homeless 2024-03-29T10:41:38.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.338+0000 7fde6e96d640 1 -- 172.21.15.84:0/2657801518 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde48005080 con 0x7fde681470b0 2024-03-29T10:41:38.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.338+0000 7fde67fff640 10 monclient: get_auth_request con 0x7fde5406df60 auth_method 0 2024-03-29T10:41:38.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.339+0000 7fde64ff9640 10 client.4429.objecter ms_handle_connect 0x7fde5406df60 2024-03-29T10:41:38.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.346+0000 7fde64ff9640 1 -- 172.21.15.84:0/2657801518 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fde60050630 con 0x7fde681470b0 2024-03-29T10:41:38.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.346+0000 7fde64ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:38.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.346+0000 7fde64ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:38.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.654+0000 7fde6e96d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:38.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.654+0000 7fde6e96d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:38.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.654+0000 7fde6e96d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:41:38.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.654+0000 7fde6e96d640 1 -- 172.21.15.84:0/2657801518 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde48005330 con 0x7fde681470b0 2024-03-29T10:41:38.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.656+0000 7fde64ff9640 1 -- 172.21.15.84:0/2657801518 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7fde60001590 con 0x7fde681470b0 2024-03-29T10:41:38.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.656+0000 7fde64ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:38.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.656+0000 7fde64ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:38.642 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:41:38.642 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:38.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.662+0000 7fde6e96d640 10 librados: watch_flush enter 2024-03-29T10:41:38.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.663+0000 7fde6e96d640 10 librados: watch_flush exit 2024-03-29T10:41:38.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.663+0000 7fde6e96d640 20 client.4429.objecter shutdown clearing up homeless session... 2024-03-29T10:41:38.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.663+0000 7fde6e96d640 10 client.4429.objecter successfully canceled tick 2024-03-29T10:41:38.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.663+0000 7fde6e96d640 1 -- 172.21.15.84:0/2657801518 >> v1:172.21.15.86:6816/3337225337 conn(0x7fde5406df60 legacy=0x7fde54070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:38.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.663+0000 7fde6e96d640 10 monclient: shutdown 2024-03-29T10:41:38.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.663+0000 7fde6e96d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:38.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.663+0000 7fde6e96d640 1 -- 172.21.15.84:0/2657801518 >> v1:172.21.15.86:6789/0 conn(0x7fde681470b0 legacy=0x7fde681dc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:38.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.663+0000 7fde6e96d640 1 -- 172.21.15.84:0/2657801518 shutdown_connections 2024-03-29T10:41:38.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.663+0000 7fde6e96d640 1 -- 172.21.15.84:0/2657801518 >> 172.21.15.84:0/2657801518 conn(0x7fde68066a00 msgr2=0x7fde681d1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:38.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.665+0000 7fde6e96d640 1 -- 172.21.15.84:0/2657801518 shutdown_connections 2024-03-29T10:41:38.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.666+0000 7fde6e96d640 1 -- 172.21.15.84:0/2657801518 wait complete. 2024-03-29T10:41:38.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:38.666+0000 7fde6e96d640 1 librados: shutdown 2024-03-29T10:41:38.667 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:45.474 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:45.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.637+0000 7f3cc4504640 10 monclient: get_monmap_and_config 2024-03-29T10:41:45.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.637+0000 7f3cc4504640 10 monclient: build_initial_monmap 2024-03-29T10:41:45.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.637+0000 7f3cc4504640 10 monclient: monmap: 2024-03-29T10:41:45.623 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:41:45.623 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:45.623 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:41:45.638867+0000 2024-03-29T10:41:45.623 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:41:45.638867+0000 2024-03-29T10:41:45.624 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:45.624 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:45.624 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:41:45.624 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:41:45.624 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:41:45.624 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:45.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.639+0000 7f3cc4504640 1 Processor -- start 2024-03-29T10:41:45.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.639+0000 7f3cc4504640 1 -- start start 2024-03-29T10:41:45.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.639+0000 7f3cc4504640 10 monclient: init 2024-03-29T10:41:45.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.640+0000 7f3cc4504640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:45.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.640+0000 7f3cc4504640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:41:45.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.640+0000 7f3cc4504640 10 monclient(hunting): picked mon.noname-c con 0x7f3cbc0a9750 addr v1:172.21.15.86:6789/0 2024-03-29T10:41:45.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.641+0000 7f3cc4504640 10 monclient(hunting): picked mon.noname-b con 0x7f3cbc149ac0 addr v1:172.21.15.84:6790/0 2024-03-29T10:41:45.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.641+0000 7f3cc1a9f640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f3cbc149ac0 0x7f3cbc149f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:45450/0 (socket says 172.21.15.84:45450) 2024-03-29T10:41:45.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.641+0000 7f3cc4504640 10 monclient(hunting): picked mon.noname-a con 0x7f3cbc14fe50 addr v1:172.21.15.84:6789/0 2024-03-29T10:41:45.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.641+0000 7f3cc4504640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cbc0aa4a0 con 0x7f3cbc14fe50 2024-03-29T10:41:45.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.641+0000 7f3cc4504640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cbc14a680 con 0x7f3cbc0a9750 2024-03-29T10:41:45.625 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.641+0000 7f3cc4504640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cbc0af290 con 0x7f3cbc149ac0 2024-03-29T10:41:45.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.641+0000 7f3cc4504640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:45.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.641+0000 7f3cc4504640 10 monclient(hunting): authenticate will time out at 1207.904691s 2024-03-29T10:41:45.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.641+0000 7f3cc1a9f640 1 -- 172.21.15.84:0/74423932 learned_addr learned my addr 172.21.15.84:0/74423932 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:41:45.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.644+0000 7f3cc129e640 1 -- 172.21.15.84:0/74423932 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 201058807 0 0) 0x7f3cbc0af290 con 0x7f3cbc149ac0 2024-03-29T10:41:45.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.644+0000 7f3cc129e640 10 monclient(hunting): _negotiate 2024-03-29T10:41:45.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.644+0000 7f3cc129e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:45.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.644+0000 7f3cc129e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:45.627 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.644+0000 7f3cc129e640 10 monclient(hunting): my global_id is 4523 2024-03-29T10:41:45.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.644+0000 7f3cc129e640 1 -- 172.21.15.84:0/74423932 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ca8003520 con 0x7f3cbc149ac0 2024-03-29T10:41:45.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.644+0000 7f3cc129e640 1 -- 172.21.15.84:0/74423932 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4123392425 0 0) 0x7f3cbc14a680 con 0x7f3cbc0a9750 2024-03-29T10:41:45.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.644+0000 7f3cc129e640 10 monclient(hunting): _negotiate 2024-03-29T10:41:45.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.644+0000 7f3cc129e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:45.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.644+0000 7f3cc129e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:45.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.644+0000 7f3cc129e640 10 monclient(hunting): my global_id is 4432 2024-03-29T10:41:45.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.644+0000 7f3cc129e640 1 -- 172.21.15.84:0/74423932 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cbc0af290 con 0x7f3cbc0a9750 2024-03-29T10:41:45.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.644+0000 7f3cc129e640 1 -- 172.21.15.84:0/74423932 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 507556558 0 0) 0x7f3cbc0af290 con 0x7f3cbc0a9750 2024-03-29T10:41:45.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.644+0000 7f3cc129e640 1 -- 172.21.15.84:0/74423932 >> v1:172.21.15.84:6790/0 conn(0x7f3cbc149ac0 legacy=0x7f3cbc149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:45.628 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.644+0000 7f3cc129e640 1 -- 172.21.15.84:0/74423932 >> v1:172.21.15.84:6789/0 conn(0x7f3cbc14fe50 legacy=0x7f3cbc152240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:41:45.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.646+0000 7f3cc129e640 10 monclient: _finish_hunting 0 2024-03-29T10:41:45.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.646+0000 7f3cc129e640 1 monclient: found mon.noname-c 2024-03-29T10:41:45.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.646+0000 7f3cc129e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:45.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.646+0000 7f3cc129e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-29T10:41:45.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.646+0000 7f3cc129e640 1 -- 172.21.15.84:0/74423932 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cbc0afb40 con 0x7f3cbc0a9750 2024-03-29T10:41:45.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.646+0000 7f3cc129e640 10 monclient: _finish_auth 0 2024-03-29T10:41:45.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.646+0000 7f3cc129e640 10 monclient: _check_auth_tickets 2024-03-29T10:41:45.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.646+0000 7f3cc129e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:45.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.646+0000 7f3cc129e640 1 -- 172.21.15.84:0/74423932 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f3cb40019f0 con 0x7f3cbc0a9750 2024-03-29T10:41:45.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.646+0000 7f3cc129e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:45.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.646+0000 7f3cc129e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:45.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.646+0000 7f3cc129e640 10 monclient: dump: 2024-03-29T10:41:45.632 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:45.632 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:45.632 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:45.632 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:45.633 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:45.633 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:45.633 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:45.633 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:45.633 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:45.633 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:45.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.646+0000 7f3cc4504640 5 monclient: authenticate success, global_id 4432 2024-03-29T10:41:45.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.646+0000 7f3cc4504640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:45.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.647+0000 7f3cc129e640 1 -- 172.21.15.84:0/74423932 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3cb4001b50 con 0x7f3cbc0a9750 2024-03-29T10:41:45.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.647+0000 7f3cc129e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:45.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.647+0000 7f3cc129e640 1 -- 172.21.15.84:0/74423932 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f3cb4001fa0 con 0x7f3cbc0a9750 2024-03-29T10:41:45.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.647+0000 7f3cc4504640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:45.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.647+0000 7f3cc4504640 4 set_mon_vals no callback set 2024-03-29T10:41:45.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.647+0000 7f3cc129e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:45.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.647+0000 7f3cc129e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:45.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.647+0000 7f3cc129e640 10 monclient: dump: 2024-03-29T10:41:45.634 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:45.634 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:45.634 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:45.634 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:45.634 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:45.634 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:45.634 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:45.634 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:45.634 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:45.634 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:45.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.648+0000 7f3cc4504640 10 monclient: shutdown 2024-03-29T10:41:45.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.648+0000 7f3cc4504640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:45.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.648+0000 7f3cc4504640 1 -- 172.21.15.84:0/74423932 >> v1:172.21.15.86:6789/0 conn(0x7f3cbc0a9750 legacy=0x7f3cbc1436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:45.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.649+0000 7f3cc4504640 1 -- 172.21.15.84:0/74423932 shutdown_connections 2024-03-29T10:41:45.635 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.649+0000 7f3cc4504640 1 -- 172.21.15.84:0/74423932 >> 172.21.15.84:0/74423932 conn(0x7f3cbc0b35e0 msgr2=0x7f3cbc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:45.636 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.650+0000 7f3cc4504640 1 -- 172.21.15.84:0/74423932 shutdown_connections 2024-03-29T10:41:45.636 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.650+0000 7f3cc4504640 1 -- 172.21.15.84:0/74423932 wait complete. 2024-03-29T10:41:45.636 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.650+0000 7f3cc4504640 10 monclient: build_initial_monmap 2024-03-29T10:41:45.636 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.650+0000 7f3cc4504640 10 monclient: monmap: 2024-03-29T10:41:45.636 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:41:45.636 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:45.636 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:41:45.637 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:41:45.637 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:45.637 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:45.637 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:41:45.637 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:41:45.637 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:41:45.637 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:45.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.651+0000 7f3cc4504640 1 librados: starting msgr at 2024-03-29T10:41:45.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.651+0000 7f3cc4504640 1 librados: starting objecter 2024-03-29T10:41:45.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.651+0000 7f3cc4504640 1 Processor -- start 2024-03-29T10:41:45.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.651+0000 7f3cc4504640 1 -- start start 2024-03-29T10:41:45.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.651+0000 7f3cc4504640 1 librados: setting wanted keys 2024-03-29T10:41:45.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.651+0000 7f3cc4504640 1 librados: calling monclient init 2024-03-29T10:41:45.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.651+0000 7f3cc4504640 10 monclient: init 2024-03-29T10:41:45.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.652+0000 7f3cc4504640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:45.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.652+0000 7f3cc4504640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:41:45.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.652+0000 7f3cc4504640 10 monclient(hunting): picked mon.noname-a con 0x7f3cbc0a9750 addr v1:172.21.15.84:6789/0 2024-03-29T10:41:45.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.652+0000 7f3cc4504640 10 monclient(hunting): picked mon.noname-b con 0x7f3cbc149ac0 addr v1:172.21.15.86:6789/0 2024-03-29T10:41:45.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.652+0000 7f3cc4504640 10 monclient(hunting): picked mon.noname-c con 0x7f3cbc14fe50 addr v1:172.21.15.84:6790/0 2024-03-29T10:41:45.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.652+0000 7f3cc4504640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cbc06ed40 con 0x7f3cbc0a9750 2024-03-29T10:41:45.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.652+0000 7f3cc4504640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cbc0710c0 con 0x7f3cbc149ac0 2024-03-29T10:41:45.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.652+0000 7f3cc4504640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cbc0713a0 con 0x7f3cbc14fe50 2024-03-29T10:41:45.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.652+0000 7f3cc4504640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:45.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.652+0000 7f3cc4504640 10 monclient(hunting): authenticate will time out at 1207.915833s 2024-03-29T10:41:45.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.654+0000 7f3cc2aa1640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f3cbc14fe50 0x7f3cbc06fa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:48210/0 (socket says 172.21.15.84:48210) 2024-03-29T10:41:45.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.654+0000 7f3cc2aa1640 1 -- 172.21.15.84:0/1206516491 learned_addr learned my addr 172.21.15.84:0/1206516491 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:41:45.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.656+0000 7f3cb2ffd640 10 client.?.objecter ms_handle_connect 0x7f3cbc14fe50 2024-03-29T10:41:45.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.656+0000 7f3cb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:45.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.656+0000 7f3cb2ffd640 1 -- 172.21.15.84:0/1206516491 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 39694991 0 0) 0x7f3cbc0713a0 con 0x7f3cbc14fe50 2024-03-29T10:41:45.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.656+0000 7f3cb2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:45.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.656+0000 7f3cb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:45.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.656+0000 7f3cb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:45.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.656+0000 7f3cb2ffd640 10 monclient(hunting): my global_id is 4526 2024-03-29T10:41:45.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.656+0000 7f3cb2ffd640 1 -- 172.21.15.84:0/1206516491 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ca0004eb0 con 0x7f3cbc14fe50 2024-03-29T10:41:45.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.656+0000 7f3cb2ffd640 1 -- 172.21.15.84:0/1206516491 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4051137043 0 0) 0x7f3ca0004eb0 con 0x7f3cbc14fe50 2024-03-29T10:41:45.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.656+0000 7f3cb2ffd640 1 -- 172.21.15.84:0/1206516491 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3cbc0713a0 con 0x7f3cbc14fe50 2024-03-29T10:41:45.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.656+0000 7f3cb2ffd640 10 client.?.objecter ms_handle_connect 0x7f3cbc149ac0 2024-03-29T10:41:45.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.656+0000 7f3cb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:45.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.656+0000 7f3cb2ffd640 1 -- 172.21.15.84:0/1206516491 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f3cb80018e0 con 0x7f3cbc14fe50 2024-03-29T10:41:45.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.656+0000 7f3cb2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:45.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.656+0000 7f3cb2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:41:45.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.656+0000 7f3cb2ffd640 10 monclient(hunting): dump: 2024-03-29T10:41:45.647 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:45.647 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:45.647 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:45.647 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:45.647 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:45.647 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:45.647 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:45.647 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:45.647 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:45.647 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:45.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.656+0000 7f3cb2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:45.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.657+0000 7f3cb2ffd640 1 -- 172.21.15.84:0/1206516491 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1430957382 0 0) 0x7f3cbc0713a0 con 0x7f3cbc14fe50 2024-03-29T10:41:45.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.657+0000 7f3cb2ffd640 1 -- 172.21.15.84:0/1206516491 >> v1:172.21.15.86:6789/0 conn(0x7f3cbc149ac0 legacy=0x7f3cbc0717c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:45.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.657+0000 7f3cb2ffd640 1 -- 172.21.15.84:0/1206516491 >> v1:172.21.15.84:6789/0 conn(0x7f3cbc0a9750 legacy=0x7f3cbc13a6f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:41:45.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.657+0000 7f3cb2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:41:45.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.657+0000 7f3cb2ffd640 1 monclient: found mon.c 2024-03-29T10:41:45.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.657+0000 7f3cb2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:45.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.657+0000 7f3cb2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:41:45.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.657+0000 7f3cb2ffd640 1 -- 172.21.15.84:0/1206516491 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cbc06efe0 con 0x7f3cbc14fe50 2024-03-29T10:41:45.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.657+0000 7f3cb2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:41:45.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.657+0000 7f3cb2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:41:45.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.657+0000 7f3cb2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:45.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.657+0000 7f3cc4504640 5 monclient: authenticate success, global_id 4526 2024-03-29T10:41:45.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.657+0000 7f3cc4504640 10 monclient: _renew_subs 2024-03-29T10:41:45.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.657+0000 7f3cc4504640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:41:45.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.657+0000 7f3cc4504640 1 -- 172.21.15.84:0/1206516491 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cbc071ed0 con 0x7f3cbc14fe50 2024-03-29T10:41:45.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.658+0000 7f3cc4504640 10 client.4526.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:45.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.658+0000 7f3cc4504640 10 monclient: _renew_subs 2024-03-29T10:41:45.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.658+0000 7f3cc4504640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:41:45.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.658+0000 7f3cc4504640 1 -- 172.21.15.84:0/1206516491 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cbc070340 con 0x7f3cbc14fe50 2024-03-29T10:41:45.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.658+0000 7f3cc4504640 1 librados: init done 2024-03-29T10:41:45.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.659+0000 7f3cb0ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:45.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.659+0000 7f3cb0ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:45.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.659+0000 7f3cb0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:41:45.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.659+0000 7f3cb0ff9640 1 -- 172.21.15.84:0/1206516491 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ca4005080 con 0x7f3cbc14fe50 2024-03-29T10:41:45.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.660+0000 7f3cb2ffd640 1 -- 172.21.15.84:0/1206516491 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3cb8001d30 con 0x7f3cbc14fe50 2024-03-29T10:41:45.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.660+0000 7f3cb2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:45.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.660+0000 7f3cb2ffd640 1 -- 172.21.15.84:0/1206516491 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f3cb80038a0 con 0x7f3cbc14fe50 2024-03-29T10:41:45.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.660+0000 7f3cb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:45.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.660+0000 7f3cb2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:45.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.660+0000 7f3cb2ffd640 10 monclient: dump: 2024-03-29T10:41:45.650 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:45.650 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:45.650 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:45.650 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:45.650 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:45.650 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:45.650 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:45.650 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:45.650 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:45.650 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:45.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.660+0000 7f3cb2ffd640 1 -- 172.21.15.84:0/1206516491 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f3cb8003b00 con 0x7f3cbc14fe50 2024-03-29T10:41:45.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.661+0000 7f3cc0a9d640 4 set_mon_vals no callback set 2024-03-29T10:41:45.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.661+0000 7f3cb2ffd640 1 -- 172.21.15.84:0/1206516491 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f3cb8084b10 con 0x7f3cbc14fe50 2024-03-29T10:41:45.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.661+0000 7f3cb2ffd640 10 client.4526.objecter ms_dispatch 0x7f3cbc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:45.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.661+0000 7f3cb2ffd640 3 client.4526.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:45.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.661+0000 7f3cb2ffd640 3 client.4526.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:45.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.661+0000 7f3cb2ffd640 20 client.4526.objecter dump_active .. 0 homeless 2024-03-29T10:41:45.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.661+0000 7f3cc22a0640 10 monclient: get_auth_request con 0x7f3ca006dcd0 auth_method 0 2024-03-29T10:41:45.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.662+0000 7f3cb2ffd640 10 client.4526.objecter ms_handle_connect 0x7f3ca006dcd0 2024-03-29T10:41:45.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.669+0000 7f3cb2ffd640 1 -- 172.21.15.84:0/1206516491 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3cb8050470 con 0x7f3cbc14fe50 2024-03-29T10:41:45.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.669+0000 7f3cb2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:45.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.669+0000 7f3cb2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:45.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.991+0000 7f3cb0ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:45.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.991+0000 7f3cb0ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:45.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.991+0000 7f3cb0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:41:45.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.991+0000 7f3cb0ff9640 1 -- 172.21.15.84:0/1206516491 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ca4005330 con 0x7f3cbc14fe50 2024-03-29T10:41:45.978 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:41:45.978 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:45.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.994+0000 7f3cb2ffd640 1 -- 172.21.15.84:0/1206516491 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f3cb80523a0 con 0x7f3cbc14fe50 2024-03-29T10:41:45.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.994+0000 7f3cb2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:45.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.994+0000 7f3cb2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:45.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.998+0000 7f3cb0ff9640 10 librados: watch_flush enter 2024-03-29T10:41:45.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.998+0000 7f3cb0ff9640 10 librados: watch_flush exit 2024-03-29T10:41:45.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.998+0000 7f3cb0ff9640 20 client.4526.objecter shutdown clearing up homeless session... 2024-03-29T10:41:45.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.998+0000 7f3cb0ff9640 10 client.4526.objecter successfully canceled tick 2024-03-29T10:41:45.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.999+0000 7f3cb0ff9640 1 -- 172.21.15.84:0/1206516491 >> v1:172.21.15.86:6816/3337225337 conn(0x7f3ca006dcd0 legacy=0x7f3ca0070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:45.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.999+0000 7f3cb0ff9640 10 monclient: shutdown 2024-03-29T10:41:45.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.999+0000 7f3cb0ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:45.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.999+0000 7f3cb0ff9640 1 -- 172.21.15.84:0/1206516491 >> v1:172.21.15.84:6790/0 conn(0x7f3cbc14fe50 legacy=0x7f3cbc06fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:45.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.999+0000 7f3cb0ff9640 1 -- 172.21.15.84:0/1206516491 shutdown_connections 2024-03-29T10:41:45.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:45.999+0000 7f3cb0ff9640 1 -- 172.21.15.84:0/1206516491 >> 172.21.15.84:0/1206516491 conn(0x7f3cbc0b35e0 msgr2=0x7f3cbc0b2370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:45.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:46.000+0000 7f3cb0ff9640 1 -- 172.21.15.84:0/1206516491 shutdown_connections 2024-03-29T10:41:45.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:46.000+0000 7f3cb0ff9640 1 -- 172.21.15.84:0/1206516491 wait complete. 2024-03-29T10:41:45.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:46.001+0000 7f3cb0ff9640 1 librados: shutdown 2024-03-29T10:41:46.000 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:41:52.807 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:41:52.955 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.971+0000 7fececce4640 10 monclient: get_monmap_and_config 2024-03-29T10:41:52.955 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.971+0000 7fececce4640 10 monclient: build_initial_monmap 2024-03-29T10:41:52.955 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.971+0000 7fececce4640 10 monclient: monmap: 2024-03-29T10:41:52.955 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:41:52.956 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:52.956 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:41:52.972881+0000 2024-03-29T10:41:52.956 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:41:52.972881+0000 2024-03-29T10:41:52.956 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:52.956 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:52.956 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:41:52.956 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:41:52.956 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:41:52.956 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:52.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.972+0000 7fececce4640 1 Processor -- start 2024-03-29T10:41:52.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.973+0000 7fececce4640 1 -- start start 2024-03-29T10:41:52.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.973+0000 7fececce4640 10 monclient: init 2024-03-29T10:41:52.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.974+0000 7fececce4640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:52.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.974+0000 7fececce4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:41:52.957 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.974+0000 7fececce4640 10 monclient(hunting): picked mon.noname-a con 0x7fece81457f0 addr v1:172.21.15.84:6789/0 2024-03-29T10:41:52.958 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.974+0000 7fececce4640 10 monclient(hunting): picked mon.noname-b con 0x7fece813f650 addr v1:172.21.15.84:6790/0 2024-03-29T10:41:52.958 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.974+0000 7fece5d9b640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fece813f650 0x7fece80b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:58654/0 (socket says 172.21.15.84:58654) 2024-03-29T10:41:52.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.975+0000 7fece5d9b640 1 -- 172.21.15.84:0/3333456641 learned_addr learned my addr 172.21.15.84:0/3333456641 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:41:52.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.975+0000 7fececce4640 10 monclient(hunting): picked mon.noname-c con 0x7fece80a9d70 addr v1:172.21.15.86:6789/0 2024-03-29T10:41:52.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.975+0000 7fececce4640 1 -- 172.21.15.84:0/3333456641 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece80aaef0 con 0x7fece81457f0 2024-03-29T10:41:52.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.975+0000 7fececce4640 1 -- 172.21.15.84:0/3333456641 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece8145bd0 con 0x7fece80a9d70 2024-03-29T10:41:52.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.975+0000 7fececce4640 1 -- 172.21.15.84:0/3333456641 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece8145eb0 con 0x7fece813f650 2024-03-29T10:41:52.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.975+0000 7fececce4640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:52.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.975+0000 7fececce4640 10 monclient(hunting): authenticate will time out at 1215.239053s 2024-03-29T10:41:52.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.976+0000 7fece559a640 1 -- 172.21.15.84:0/3333456641 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569121041 0 0) 0x7fece8145eb0 con 0x7fece813f650 2024-03-29T10:41:52.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.976+0000 7fece559a640 10 monclient(hunting): _negotiate 2024-03-29T10:41:52.959 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.976+0000 7fece559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:52.960 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.976+0000 7fece559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:52.960 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.976+0000 7fece559a640 10 monclient(hunting): my global_id is 4529 2024-03-29T10:41:52.960 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.976+0000 7fece559a640 1 -- 172.21.15.84:0/3333456641 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecd0003520 con 0x7fece813f650 2024-03-29T10:41:52.960 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.976+0000 7fece559a640 1 -- 172.21.15.84:0/3333456641 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 755941753 0 0) 0x7fecd0003520 con 0x7fece813f650 2024-03-29T10:41:52.960 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.976+0000 7fece559a640 1 -- 172.21.15.84:0/3333456641 >> v1:172.21.15.86:6789/0 conn(0x7fece80a9d70 legacy=0x7fece814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:41:52.960 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.977+0000 7fece559a640 1 -- 172.21.15.84:0/3333456641 >> v1:172.21.15.84:6789/0 conn(0x7fece81457f0 legacy=0x7fece813ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:41:52.962 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.978+0000 7fece559a640 10 monclient: _finish_hunting 0 2024-03-29T10:41:52.962 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.978+0000 7fece559a640 1 monclient: found mon.noname-b 2024-03-29T10:41:52.962 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.978+0000 7fece559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:52.962 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.978+0000 7fece559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:41:52.962 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.978+0000 7fece559a640 1 -- 172.21.15.84:0/3333456641 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece8146340 con 0x7fece813f650 2024-03-29T10:41:52.962 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.978+0000 7fece559a640 10 monclient: _finish_auth 0 2024-03-29T10:41:52.962 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.978+0000 7fece559a640 10 monclient: _check_auth_tickets 2024-03-29T10:41:52.963 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.978+0000 7fece559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:52.963 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.978+0000 7fece559a640 1 -- 172.21.15.84:0/3333456641 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fecdc001690 con 0x7fece813f650 2024-03-29T10:41:52.963 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.978+0000 7fece559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:52.963 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.978+0000 7fece559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:52.963 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.978+0000 7fece559a640 10 monclient: dump: 2024-03-29T10:41:52.963 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:52.963 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:52.963 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:52.963 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:52.963 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:52.963 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:52.963 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:52.963 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:52.963 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:52.964 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:52.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.978+0000 7fececce4640 5 monclient: authenticate success, global_id 4529 2024-03-29T10:41:52.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.978+0000 7fececce4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:41:52.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.978+0000 7fece559a640 1 -- 172.21.15.84:0/3333456641 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fecdc001920 con 0x7fece813f650 2024-03-29T10:41:52.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.978+0000 7fece559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:52.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.978+0000 7fece559a640 1 -- 172.21.15.84:0/3333456641 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fecdc001cb0 con 0x7fece813f650 2024-03-29T10:41:52.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.978+0000 7fece559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:52.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.978+0000 7fece559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:52.964 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.978+0000 7fece559a640 10 monclient: dump: 2024-03-29T10:41:52.964 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:52.964 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:52.964 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:52.964 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:52.964 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:52.964 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:52.965 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:52.965 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:52.965 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:52.965 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:52.966 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.979+0000 7fececce4640 10 monclient: get_monmap_and_config success 2024-03-29T10:41:52.966 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.979+0000 7fececce4640 4 set_mon_vals no callback set 2024-03-29T10:41:52.966 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.980+0000 7fececce4640 10 monclient: shutdown 2024-03-29T10:41:52.966 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.980+0000 7fececce4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:52.966 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.980+0000 7fececce4640 1 -- 172.21.15.84:0/3333456641 >> v1:172.21.15.84:6790/0 conn(0x7fece813f650 legacy=0x7fece80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:52.966 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.982+0000 7fececce4640 1 -- 172.21.15.84:0/3333456641 shutdown_connections 2024-03-29T10:41:52.966 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.982+0000 7fececce4640 1 -- 172.21.15.84:0/3333456641 >> 172.21.15.84:0/3333456641 conn(0x7fece8066a00 msgr2=0x7fece8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:52.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.983+0000 7fececce4640 1 -- 172.21.15.84:0/3333456641 shutdown_connections 2024-03-29T10:41:52.967 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.983+0000 7fececce4640 1 -- 172.21.15.84:0/3333456641 wait complete. 2024-03-29T10:41:52.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.984+0000 7fececce4640 10 monclient: build_initial_monmap 2024-03-29T10:41:52.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.984+0000 7fececce4640 10 monclient: monmap: 2024-03-29T10:41:52.970 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:41:52.970 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:41:52.970 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:41:52.970 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:41:52.970 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:41:52.970 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:52.970 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:41:52.970 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:41:52.970 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:41:52.970 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:52.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.984+0000 7fececce4640 1 librados: starting msgr at 2024-03-29T10:41:52.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.984+0000 7fececce4640 1 librados: starting objecter 2024-03-29T10:41:52.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.985+0000 7fececce4640 1 Processor -- start 2024-03-29T10:41:52.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.985+0000 7fececce4640 1 -- start start 2024-03-29T10:41:52.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.985+0000 7fececce4640 1 librados: setting wanted keys 2024-03-29T10:41:52.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.985+0000 7fececce4640 1 librados: calling monclient init 2024-03-29T10:41:52.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.985+0000 7fececce4640 10 monclient: init 2024-03-29T10:41:52.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.985+0000 7fececce4640 10 monclient: _reopen_session rank -1 2024-03-29T10:41:52.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.985+0000 7fececce4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:41:52.971 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.985+0000 7fececce4640 10 monclient(hunting): picked mon.noname-c con 0x7fece80a9d70 addr v1:172.21.15.84:6790/0 2024-03-29T10:41:52.972 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.988+0000 7fececce4640 10 monclient(hunting): picked mon.noname-b con 0x7fece813f650 addr v1:172.21.15.86:6789/0 2024-03-29T10:41:52.972 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.988+0000 7fececce4640 10 monclient(hunting): picked mon.noname-a con 0x7fece81457f0 addr v1:172.21.15.84:6789/0 2024-03-29T10:41:52.972 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.988+0000 7fececce4640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece8134a70 con 0x7fece81457f0 2024-03-29T10:41:52.972 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.988+0000 7fececce4640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece8134ce0 con 0x7fece813f650 2024-03-29T10:41:52.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.988+0000 7fececce4640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece8132ae0 con 0x7fece80a9d70 2024-03-29T10:41:52.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.988+0000 7fececce4640 10 monclient(hunting): _renew_subs 2024-03-29T10:41:52.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.988+0000 7fececce4640 10 monclient(hunting): authenticate will time out at 1215.251378s 2024-03-29T10:41:52.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.988+0000 7fece659c640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fece80a9d70 0x7fece8133400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:59456/0 (socket says 172.21.15.84:59456) 2024-03-29T10:41:52.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.988+0000 7fece659c640 1 -- 172.21.15.84:0/3725384983 learned_addr learned my addr 172.21.15.84:0/3725384983 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:41:52.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.990+0000 7fecd6ffd640 10 client.?.objecter ms_handle_connect 0x7fece80a9d70 2024-03-29T10:41:52.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.990+0000 7fecd6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:52.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.990+0000 7fecd6ffd640 10 client.?.objecter ms_handle_connect 0x7fece813f650 2024-03-29T10:41:52.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.990+0000 7fecd6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:41:52.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.990+0000 7fecd6ffd640 1 -- 172.21.15.84:0/3725384983 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924851668 0 0) 0x7fece8132ae0 con 0x7fece80a9d70 2024-03-29T10:41:52.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.990+0000 7fecd6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:52.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.990+0000 7fecd6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:52.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.990+0000 7fecd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:52.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.990+0000 7fecd6ffd640 10 monclient(hunting): my global_id is 4532 2024-03-29T10:41:52.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.990+0000 7fecd6ffd640 1 -- 172.21.15.84:0/3725384983 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecc8004eb0 con 0x7fece80a9d70 2024-03-29T10:41:52.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.990+0000 7fecd6ffd640 1 -- 172.21.15.84:0/3725384983 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2049356280 0 0) 0x7fece8134ce0 con 0x7fece813f650 2024-03-29T10:41:52.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.990+0000 7fecd6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:41:52.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.990+0000 7fecd6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:41:52.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.990+0000 7fecd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:41:52.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.990+0000 7fecd6ffd640 10 monclient(hunting): my global_id is 4438 2024-03-29T10:41:52.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.990+0000 7fecd6ffd640 1 -- 172.21.15.84:0/3725384983 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fece8132ae0 con 0x7fece813f650 2024-03-29T10:41:52.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.991+0000 7fecd6ffd640 1 -- 172.21.15.84:0/3725384983 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1985781687 0 0) 0x7fece8132ae0 con 0x7fece813f650 2024-03-29T10:41:52.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.991+0000 7fecd6ffd640 1 -- 172.21.15.84:0/3725384983 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fece8134ce0 con 0x7fece813f650 2024-03-29T10:41:52.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.991+0000 7fecd6ffd640 1 -- 172.21.15.84:0/3725384983 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2273355394 0 0) 0x7fecc8004eb0 con 0x7fece80a9d70 2024-03-29T10:41:52.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.991+0000 7fecd6ffd640 1 -- 172.21.15.84:0/3725384983 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fece8132ae0 con 0x7fece80a9d70 2024-03-29T10:41:52.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.991+0000 7fecd6ffd640 1 -- 172.21.15.84:0/3725384983 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fecdc001550 con 0x7fece813f650 2024-03-29T10:41:52.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.991+0000 7fecd6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:52.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.991+0000 7fecd6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:41:52.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.991+0000 7fecd6ffd640 10 monclient(hunting): dump: 2024-03-29T10:41:52.976 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:52.976 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:52.976 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:52.976 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:52.976 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:52.976 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:52.976 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:52.977 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:52.977 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:52.977 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:52.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.991+0000 7fecd6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:41:52.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.991+0000 7fecd6ffd640 1 -- 172.21.15.84:0/3725384983 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fecd8001750 con 0x7fece80a9d70 2024-03-29T10:41:52.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.991+0000 7fecd6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:52.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.991+0000 7fecd6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:41:52.977 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.991+0000 7fecd6ffd640 10 monclient(hunting): dump: 2024-03-29T10:41:52.977 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:52.977 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:52.977 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:52.977 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:52.977 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:52.977 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:52.978 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:52.978 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:52.978 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:52.978 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:52.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.991+0000 7fecd6ffd640 1 -- 172.21.15.84:0/3725384983 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 810981795 0 0) 0x7fece8134ce0 con 0x7fece813f650 2024-03-29T10:41:52.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.991+0000 7fecd6ffd640 1 -- 172.21.15.84:0/3725384983 >> v1:172.21.15.84:6790/0 conn(0x7fece80a9d70 legacy=0x7fece8133400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:52.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.991+0000 7fecd6ffd640 1 -- 172.21.15.84:0/3725384983 >> v1:172.21.15.84:6789/0 conn(0x7fece81457f0 legacy=0x7fece812d8b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:41:52.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.991+0000 7fecd6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:41:52.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.991+0000 7fecd6ffd640 1 monclient: found mon.b 2024-03-29T10:41:52.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.991+0000 7fecd6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:41:52.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.991+0000 7fecd6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:41:52.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.991+0000 7fecd6ffd640 1 -- 172.21.15.84:0/3725384983 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece8133b10 con 0x7fece813f650 2024-03-29T10:41:52.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.992+0000 7fecd6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:41:52.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.992+0000 7fecd6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:41:52.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.992+0000 7fecd6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:41:52.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.992+0000 7fececce4640 5 monclient: authenticate success, global_id 4438 2024-03-29T10:41:52.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.992+0000 7fececce4640 10 monclient: _renew_subs 2024-03-29T10:41:52.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.992+0000 7fececce4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:41:52.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.992+0000 7fececce4640 1 -- 172.21.15.84:0/3725384983 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fece808bc10 con 0x7fece813f650 2024-03-29T10:41:52.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.992+0000 7fececce4640 10 client.4438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:41:52.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.992+0000 7fececce4640 10 monclient: _renew_subs 2024-03-29T10:41:52.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.992+0000 7fececce4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:41:52.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.992+0000 7fececce4640 1 -- 172.21.15.84:0/3725384983 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fece8141d40 con 0x7fece813f650 2024-03-29T10:41:52.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.992+0000 7fececce4640 1 librados: init done 2024-03-29T10:41:52.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.993+0000 7fecd6ffd640 1 -- 172.21.15.84:0/3725384983 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fecdc0016b0 con 0x7fece813f650 2024-03-29T10:41:52.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.993+0000 7fecd6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:41:52.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.993+0000 7fecd6ffd640 1 -- 172.21.15.84:0/3725384983 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fecdc001250 con 0x7fece813f650 2024-03-29T10:41:52.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.993+0000 7fecd6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:41:52.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.993+0000 7fecd6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:41:52.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.993+0000 7fecd6ffd640 10 monclient: dump: 2024-03-29T10:41:52.981 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:41:52.981 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:41:52.981 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:52.981 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:41:52.981 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:41:52.982 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:41:52.982 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:41:52.982 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:41:52.982 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:41:52.982 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:41:52.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.993+0000 7fecd7fff640 4 set_mon_vals no callback set 2024-03-29T10:41:52.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.993+0000 7fececce4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:52.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.994+0000 7fececce4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:52.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.994+0000 7fececce4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:41:52.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.994+0000 7fececce4640 1 -- 172.21.15.84:0/3725384983 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feccc005080 con 0x7fece813f650 2024-03-29T10:41:52.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.995+0000 7fecd6ffd640 1 -- 172.21.15.84:0/3725384983 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7fecdc01ac00 con 0x7fece813f650 2024-03-29T10:41:52.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.996+0000 7fecd6ffd640 1 -- 172.21.15.84:0/3725384983 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7fecdc084a20 con 0x7fece813f650 2024-03-29T10:41:52.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.996+0000 7fecd6ffd640 10 client.4438.objecter ms_dispatch 0x7fece8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:41:52.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.996+0000 7fecd6ffd640 3 client.4438.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:41:52.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.996+0000 7fecd6ffd640 3 client.4438.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:41:52.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.996+0000 7fecd6ffd640 20 client.4438.objecter dump_active .. 0 homeless 2024-03-29T10:41:52.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:52.997+0000 7fece659c640 10 monclient: get_auth_request con 0x7fecc806df60 auth_method 0 2024-03-29T10:41:52.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:53.000+0000 7fecd6ffd640 10 client.4438.objecter ms_handle_connect 0x7fecc806df60 2024-03-29T10:41:52.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:53.003+0000 7fecd6ffd640 1 -- 172.21.15.84:0/3725384983 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fecdc050130 con 0x7fece813f650 2024-03-29T10:41:52.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:53.003+0000 7fecd6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:41:52.987 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:53.003+0000 7fecd6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:41:53.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:53.332+0000 7fececce4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:53.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:53.332+0000 7fececce4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:53.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:53.332+0000 7fececce4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:41:53.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:53.332+0000 7fececce4640 1 -- 172.21.15.84:0/3725384983 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feccc005330 con 0x7fece813f650 2024-03-29T10:41:53.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:53.333+0000 7fecd6ffd640 1 -- 172.21.15.84:0/3725384983 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7fecdc052230 con 0x7fece813f650 2024-03-29T10:41:53.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:53.333+0000 7fecd6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:41:53.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:53.333+0000 7fecd6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:41:53.319 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:41:53.319 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:41:53.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:53.337+0000 7fececce4640 10 librados: watch_flush enter 2024-03-29T10:41:53.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:53.337+0000 7fececce4640 10 librados: watch_flush exit 2024-03-29T10:41:53.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:53.337+0000 7fececce4640 20 client.4438.objecter shutdown clearing up homeless session... 2024-03-29T10:41:53.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:53.337+0000 7fececce4640 10 client.4438.objecter successfully canceled tick 2024-03-29T10:41:53.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:53.338+0000 7fececce4640 1 -- 172.21.15.84:0/3725384983 >> v1:172.21.15.86:6816/3337225337 conn(0x7fecc806df60 legacy=0x7fecc8070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:53.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:53.338+0000 7fececce4640 10 monclient: shutdown 2024-03-29T10:41:53.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:53.338+0000 7fececce4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:41:53.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:53.338+0000 7fececce4640 1 -- 172.21.15.84:0/3725384983 >> v1:172.21.15.86:6789/0 conn(0x7fece813f650 legacy=0x7fece8141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:41:53.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:53.338+0000 7fececce4640 1 -- 172.21.15.84:0/3725384983 shutdown_connections 2024-03-29T10:41:53.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:53.338+0000 7fececce4640 1 -- 172.21.15.84:0/3725384983 >> 172.21.15.84:0/3725384983 conn(0x7fece8066a00 msgr2=0x7fece80b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:41:53.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:53.338+0000 7fececce4640 1 -- 172.21.15.84:0/3725384983 shutdown_connections 2024-03-29T10:41:53.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:53.338+0000 7fececce4640 1 -- 172.21.15.84:0/3725384983 wait complete. 2024-03-29T10:41:53.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:41:53.338+0000 7fececce4640 1 librados: shutdown 2024-03-29T10:41:53.337 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:00.144 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:00.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.309+0000 7fbe5452b640 10 monclient: get_monmap_and_config 2024-03-29T10:42:00.293 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.309+0000 7fbe5452b640 10 monclient: build_initial_monmap 2024-03-29T10:42:00.294 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.309+0000 7fbe5452b640 10 monclient: monmap: 2024-03-29T10:42:00.294 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:42:00.294 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:00.294 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:42:00.310912+0000 2024-03-29T10:42:00.294 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:42:00.310912+0000 2024-03-29T10:42:00.294 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:00.294 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:00.294 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:42:00.294 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:42:00.295 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:42:00.295 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:00.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.310+0000 7fbe5452b640 1 Processor -- start 2024-03-29T10:42:00.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.310+0000 7fbe5452b640 1 -- start start 2024-03-29T10:42:00.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.310+0000 7fbe5452b640 10 monclient: init 2024-03-29T10:42:00.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.311+0000 7fbe5452b640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:00.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.311+0000 7fbe5452b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:42:00.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.311+0000 7fbe5452b640 10 monclient(hunting): picked mon.noname-b con 0x7fbe4c0b0770 addr v1:172.21.15.84:6790/0 2024-03-29T10:42:00.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.311+0000 7fbe5452b640 10 monclient(hunting): picked mon.noname-c con 0x7fbe4c14a700 addr v1:172.21.15.86:6789/0 2024-03-29T10:42:00.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.312+0000 7fbe5452b640 10 monclient(hunting): picked mon.noname-a con 0x7fbe4c0ac5a0 addr v1:172.21.15.84:6789/0 2024-03-29T10:42:00.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.312+0000 7fbe5452b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe4c0aba70 con 0x7fbe4c0ac5a0 2024-03-29T10:42:00.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.312+0000 7fbe5452b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe4c0aecb0 con 0x7fbe4c14a700 2024-03-29T10:42:00.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.312+0000 7fbe5452b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe4c0af150 con 0x7fbe4c0b0770 2024-03-29T10:42:00.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.312+0000 7fbe5452b640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:00.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.312+0000 7fbe5452b640 10 monclient(hunting): authenticate will time out at 1222.575148s 2024-03-29T10:42:00.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.313+0000 7fbe522c7640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fbe4c0b0770 0x7fbe4c149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:53740/0 (socket says 172.21.15.84:53740) 2024-03-29T10:42:00.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.313+0000 7fbe522c7640 1 -- 172.21.15.84:0/4129281583 learned_addr learned my addr 172.21.15.84:0/4129281583 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:42:00.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.315+0000 7fbe512c5640 1 -- 172.21.15.84:0/4129281583 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1211545663 0 0) 0x7fbe4c0af150 con 0x7fbe4c0b0770 2024-03-29T10:42:00.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.315+0000 7fbe512c5640 10 monclient(hunting): _negotiate 2024-03-29T10:42:00.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.315+0000 7fbe512c5640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:00.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.315+0000 7fbe512c5640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:00.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.315+0000 7fbe512c5640 10 monclient(hunting): my global_id is 4535 2024-03-29T10:42:00.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.315+0000 7fbe512c5640 1 -- 172.21.15.84:0/4129281583 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe38003520 con 0x7fbe4c0b0770 2024-03-29T10:42:00.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.316+0000 7fbe512c5640 1 -- 172.21.15.84:0/4129281583 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3979715717 0 0) 0x7fbe38003520 con 0x7fbe4c0b0770 2024-03-29T10:42:00.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.316+0000 7fbe512c5640 1 -- 172.21.15.84:0/4129281583 >> v1:172.21.15.86:6789/0 conn(0x7fbe4c14a700 legacy=0x7fbe4c0abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:00.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.317+0000 7fbe512c5640 1 -- 172.21.15.84:0/4129281583 >> v1:172.21.15.84:6789/0 conn(0x7fbe4c0ac5a0 legacy=0x7fbe4c151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:00.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.319+0000 7fbe512c5640 10 monclient: _finish_hunting 0 2024-03-29T10:42:00.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.319+0000 7fbe512c5640 1 monclient: found mon.noname-b 2024-03-29T10:42:00.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.319+0000 7fbe512c5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:00.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.319+0000 7fbe512c5640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:42:00.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.319+0000 7fbe512c5640 1 -- 172.21.15.84:0/4129281583 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe4c0ac9d0 con 0x7fbe4c0b0770 2024-03-29T10:42:00.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.319+0000 7fbe512c5640 10 monclient: _finish_auth 0 2024-03-29T10:42:00.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.319+0000 7fbe512c5640 10 monclient: _check_auth_tickets 2024-03-29T10:42:00.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.319+0000 7fbe512c5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:00.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.319+0000 7fbe512c5640 1 -- 172.21.15.84:0/4129281583 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fbe48001690 con 0x7fbe4c0b0770 2024-03-29T10:42:00.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.319+0000 7fbe512c5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:00.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.319+0000 7fbe512c5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:00.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.319+0000 7fbe512c5640 10 monclient: dump: 2024-03-29T10:42:00.304 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:00.304 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:00.304 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:00.304 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:00.304 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:00.304 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:00.304 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:00.304 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:00.304 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:00.304 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:00.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.320+0000 7fbe512c5640 1 -- 172.21.15.84:0/4129281583 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe48001920 con 0x7fbe4c0b0770 2024-03-29T10:42:00.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.320+0000 7fbe512c5640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:00.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.320+0000 7fbe512c5640 1 -- 172.21.15.84:0/4129281583 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fbe48001cb0 con 0x7fbe4c0b0770 2024-03-29T10:42:00.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.320+0000 7fbe512c5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:00.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.320+0000 7fbe512c5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:00.305 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.320+0000 7fbe512c5640 10 monclient: dump: 2024-03-29T10:42:00.305 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:00.305 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:00.305 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:00.305 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:00.305 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:00.305 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:00.305 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:00.305 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:00.305 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:00.305 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:00.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.320+0000 7fbe5452b640 5 monclient: authenticate success, global_id 4535 2024-03-29T10:42:00.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.320+0000 7fbe5452b640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:00.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.320+0000 7fbe5452b640 4 set_mon_vals no callback set 2024-03-29T10:42:00.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.321+0000 7fbe5452b640 10 monclient: shutdown 2024-03-29T10:42:00.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.321+0000 7fbe5452b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:00.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.321+0000 7fbe5452b640 1 -- 172.21.15.84:0/4129281583 >> v1:172.21.15.84:6790/0 conn(0x7fbe4c0b0770 legacy=0x7fbe4c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:00.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.321+0000 7fbe5452b640 1 -- 172.21.15.84:0/4129281583 shutdown_connections 2024-03-29T10:42:00.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.321+0000 7fbe5452b640 1 -- 172.21.15.84:0/4129281583 >> 172.21.15.84:0/4129281583 conn(0x7fbe4c06b660 msgr2=0x7fbe4c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:00.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.321+0000 7fbe5452b640 1 -- 172.21.15.84:0/4129281583 shutdown_connections 2024-03-29T10:42:00.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.321+0000 7fbe5452b640 1 -- 172.21.15.84:0/4129281583 wait complete. 2024-03-29T10:42:00.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.322+0000 7fbe5452b640 10 monclient: build_initial_monmap 2024-03-29T10:42:00.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.322+0000 7fbe5452b640 10 monclient: monmap: 2024-03-29T10:42:00.309 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:42:00.309 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:00.309 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:42:00.309 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:42:00.309 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:00.309 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:00.309 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:42:00.309 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:42:00.309 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:42:00.309 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:00.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.323+0000 7fbe5452b640 1 librados: starting msgr at 2024-03-29T10:42:00.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.323+0000 7fbe5452b640 1 librados: starting objecter 2024-03-29T10:42:00.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.323+0000 7fbe5452b640 1 Processor -- start 2024-03-29T10:42:00.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.323+0000 7fbe5452b640 1 -- start start 2024-03-29T10:42:00.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.323+0000 7fbe5452b640 1 librados: setting wanted keys 2024-03-29T10:42:00.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.323+0000 7fbe5452b640 1 librados: calling monclient init 2024-03-29T10:42:00.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.323+0000 7fbe5452b640 10 monclient: init 2024-03-29T10:42:00.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.324+0000 7fbe5452b640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:00.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.324+0000 7fbe5452b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:42:00.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.324+0000 7fbe5452b640 10 monclient(hunting): picked mon.noname-a con 0x7fbe4c0ac5a0 addr v1:172.21.15.84:6789/0 2024-03-29T10:42:00.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.327+0000 7fbe5452b640 10 monclient(hunting): picked mon.noname-b con 0x7fbe4c0b0770 addr v1:172.21.15.86:6789/0 2024-03-29T10:42:00.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.327+0000 7fbe5452b640 10 monclient(hunting): picked mon.noname-c con 0x7fbe4c14a700 addr v1:172.21.15.84:6790/0 2024-03-29T10:42:00.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.327+0000 7fbe5452b640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe4c1239d0 con 0x7fbe4c0ac5a0 2024-03-29T10:42:00.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.327+0000 7fbe5452b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe4c127f90 con 0x7fbe4c0b0770 2024-03-29T10:42:00.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.328+0000 7fbe5452b640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe4c128200 con 0x7fbe4c14a700 2024-03-29T10:42:00.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.328+0000 7fbe5452b640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:00.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.328+0000 7fbe5452b640 10 monclient(hunting): authenticate will time out at 1222.591182s 2024-03-29T10:42:00.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.328+0000 7fbe51ac6640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fbe4c0b0770 0x7fbe4c144df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:47411/0 (socket says 172.21.15.84:47411) 2024-03-29T10:42:00.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.328+0000 7fbe51ac6640 1 -- 172.21.15.84:0/4153900279 learned_addr learned my addr 172.21.15.84:0/4153900279 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:42:00.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.328+0000 7fbe42ffd640 10 client.?.objecter ms_handle_connect 0x7fbe4c0b0770 2024-03-29T10:42:00.314 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.328+0000 7fbe42ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:00.314 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.328+0000 7fbe42ffd640 1 -- 172.21.15.84:0/4153900279 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2583237774 0 0) 0x7fbe4c127f90 con 0x7fbe4c0b0770 2024-03-29T10:42:00.314 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.328+0000 7fbe42ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:00.314 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.328+0000 7fbe42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:00.314 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.328+0000 7fbe42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:00.314 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.328+0000 7fbe42ffd640 10 monclient(hunting): my global_id is 4441 2024-03-29T10:42:00.314 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.328+0000 7fbe42ffd640 1 -- 172.21.15.84:0/4153900279 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe30004eb0 con 0x7fbe4c0b0770 2024-03-29T10:42:00.314 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.329+0000 7fbe42ffd640 1 -- 172.21.15.84:0/4153900279 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1059492311 0 0) 0x7fbe30004eb0 con 0x7fbe4c0b0770 2024-03-29T10:42:00.314 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.329+0000 7fbe42ffd640 1 -- 172.21.15.84:0/4153900279 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe4c127f90 con 0x7fbe4c0b0770 2024-03-29T10:42:00.314 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.329+0000 7fbe42ffd640 1 -- 172.21.15.84:0/4153900279 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fbe3c0017d0 con 0x7fbe4c0b0770 2024-03-29T10:42:00.314 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.329+0000 7fbe42ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:00.314 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.329+0000 7fbe42ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:00.314 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.329+0000 7fbe42ffd640 10 monclient(hunting): dump: 2024-03-29T10:42:00.314 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:00.315 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:00.315 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:00.315 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:00.315 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:00.315 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:00.315 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:00.315 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:00.315 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:00.315 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:00.315 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.329+0000 7fbe42ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:00.315 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.329+0000 7fbe42ffd640 10 client.?.objecter ms_handle_connect 0x7fbe4c14a700 2024-03-29T10:42:00.315 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.329+0000 7fbe42ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:00.315 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.329+0000 7fbe42ffd640 1 -- 172.21.15.84:0/4153900279 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818389787 0 0) 0x7fbe4c128200 con 0x7fbe4c14a700 2024-03-29T10:42:00.315 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.329+0000 7fbe42ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:00.315 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.329+0000 7fbe42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:00.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.329+0000 7fbe42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:00.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.329+0000 7fbe42ffd640 10 monclient(hunting): my global_id is 4538 2024-03-29T10:42:00.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.329+0000 7fbe42ffd640 1 -- 172.21.15.84:0/4153900279 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe30004eb0 con 0x7fbe4c14a700 2024-03-29T10:42:00.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.329+0000 7fbe42ffd640 1 -- 172.21.15.84:0/4153900279 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 805963901 0 0) 0x7fbe4c127f90 con 0x7fbe4c0b0770 2024-03-29T10:42:00.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.330+0000 7fbe42ffd640 1 -- 172.21.15.84:0/4153900279 >> v1:172.21.15.84:6790/0 conn(0x7fbe4c14a700 legacy=0x7fbe4c1db910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:00.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.330+0000 7fbe42ffd640 1 -- 172.21.15.84:0/4153900279 >> v1:172.21.15.84:6789/0 conn(0x7fbe4c0ac5a0 legacy=0x7fbe4c126c50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:42:00.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.331+0000 7fbe42ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:42:00.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.331+0000 7fbe42ffd640 1 monclient: found mon.b 2024-03-29T10:42:00.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.331+0000 7fbe42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:00.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.331+0000 7fbe42ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:42:00.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.331+0000 7fbe42ffd640 1 -- 172.21.15.84:0/4153900279 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe4c125e80 con 0x7fbe4c0b0770 2024-03-29T10:42:00.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.331+0000 7fbe42ffd640 10 monclient: _finish_auth 0 2024-03-29T10:42:00.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.331+0000 7fbe42ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:42:00.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.331+0000 7fbe42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:00.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.331+0000 7fbe5452b640 5 monclient: authenticate success, global_id 4441 2024-03-29T10:42:00.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.331+0000 7fbe5452b640 10 monclient: _renew_subs 2024-03-29T10:42:00.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.331+0000 7fbe5452b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:42:00.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.331+0000 7fbe5452b640 1 -- 172.21.15.84:0/4153900279 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe4c145b00 con 0x7fbe4c0b0770 2024-03-29T10:42:00.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.331+0000 7fbe5452b640 10 client.4441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:00.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.331+0000 7fbe5452b640 10 monclient: _renew_subs 2024-03-29T10:42:00.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.331+0000 7fbe5452b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:42:00.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.331+0000 7fbe5452b640 1 -- 172.21.15.84:0/4153900279 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe4c1456e0 con 0x7fbe4c0b0770 2024-03-29T10:42:00.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.331+0000 7fbe5452b640 1 librados: init done 2024-03-29T10:42:00.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.332+0000 7fbe42ffd640 1 -- 172.21.15.84:0/4153900279 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe3c001930 con 0x7fbe4c0b0770 2024-03-29T10:42:00.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.332+0000 7fbe42ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:00.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.332+0000 7fbe50ac4640 4 set_mon_vals no callback set 2024-03-29T10:42:00.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.334+0000 7fbe42ffd640 1 -- 172.21.15.84:0/4153900279 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fbe3c004a40 con 0x7fbe4c0b0770 2024-03-29T10:42:00.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.334+0000 7fbe42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:00.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.334+0000 7fbe42ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:00.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.334+0000 7fbe42ffd640 10 monclient: dump: 2024-03-29T10:42:00.320 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:00.320 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:00.320 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:00.320 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:00.320 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:00.320 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:00.320 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:00.320 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:00.320 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:00.320 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:00.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.335+0000 7fbe40ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:00.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.335+0000 7fbe40ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:00.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.335+0000 7fbe40ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:42:00.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.335+0000 7fbe40ff9640 1 -- 172.21.15.84:0/4153900279 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe4c1273f0 con 0x7fbe4c0b0770 2024-03-29T10:42:00.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.335+0000 7fbe42ffd640 1 -- 172.21.15.84:0/4153900279 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7fbe3c01aa90 con 0x7fbe4c0b0770 2024-03-29T10:42:00.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.336+0000 7fbe42ffd640 1 -- 172.21.15.84:0/4153900279 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7fbe3c084ae0 con 0x7fbe4c0b0770 2024-03-29T10:42:00.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.336+0000 7fbe42ffd640 10 client.4441.objecter ms_dispatch 0x7fbe4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:00.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.336+0000 7fbe42ffd640 3 client.4441.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:00.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.336+0000 7fbe42ffd640 3 client.4441.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:00.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.336+0000 7fbe42ffd640 20 client.4441.objecter dump_active .. 0 homeless 2024-03-29T10:42:00.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.337+0000 7fbe522c7640 10 monclient: get_auth_request con 0x7fbe3006dd00 auth_method 0 2024-03-29T10:42:00.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.337+0000 7fbe42ffd640 10 client.4441.objecter ms_handle_connect 0x7fbe3006dd00 2024-03-29T10:42:00.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.343+0000 7fbe42ffd640 1 -- 172.21.15.84:0/4153900279 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbe4c1273f0 con 0x7fbe4c0b0770 2024-03-29T10:42:00.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.344+0000 7fbe42ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:00.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.344+0000 7fbe42ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:00.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.659+0000 7fbe40ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:00.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.659+0000 7fbe40ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:00.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.659+0000 7fbe40ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:42:00.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.659+0000 7fbe40ff9640 1 -- 172.21.15.84:0/4153900279 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe4c1273f0 con 0x7fbe4c0b0770 2024-03-29T10:42:00.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.661+0000 7fbe42ffd640 1 -- 172.21.15.84:0/4153900279 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7fbe4c1273f0 con 0x7fbe4c0b0770 2024-03-29T10:42:00.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.661+0000 7fbe42ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:00.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.661+0000 7fbe42ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:00.645 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:42:00.645 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:00.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.664+0000 7fbe40ff9640 10 librados: watch_flush enter 2024-03-29T10:42:00.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.664+0000 7fbe40ff9640 10 librados: watch_flush exit 2024-03-29T10:42:00.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.664+0000 7fbe40ff9640 20 client.4441.objecter shutdown clearing up homeless session... 2024-03-29T10:42:00.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.664+0000 7fbe40ff9640 10 client.4441.objecter successfully canceled tick 2024-03-29T10:42:00.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.664+0000 7fbe40ff9640 1 -- 172.21.15.84:0/4153900279 >> v1:172.21.15.86:6816/3337225337 conn(0x7fbe3006dd00 legacy=0x7fbe300701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:00.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.664+0000 7fbe40ff9640 10 monclient: shutdown 2024-03-29T10:42:00.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.664+0000 7fbe40ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:00.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.664+0000 7fbe40ff9640 1 -- 172.21.15.84:0/4153900279 >> v1:172.21.15.86:6789/0 conn(0x7fbe4c0b0770 legacy=0x7fbe4c144df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:00.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.665+0000 7fbe40ff9640 1 -- 172.21.15.84:0/4153900279 shutdown_connections 2024-03-29T10:42:00.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.665+0000 7fbe40ff9640 1 -- 172.21.15.84:0/4153900279 >> 172.21.15.84:0/4153900279 conn(0x7fbe4c06b660 msgr2=0x7fbe4c0b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:00.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.666+0000 7fbe40ff9640 1 -- 172.21.15.84:0/4153900279 shutdown_connections 2024-03-29T10:42:00.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.666+0000 7fbe40ff9640 1 -- 172.21.15.84:0/4153900279 wait complete. 2024-03-29T10:42:00.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:00.666+0000 7fbe40ff9640 1 librados: shutdown 2024-03-29T10:42:00.666 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:07.471 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:07.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.638+0000 7f4db60c1640 10 monclient: get_monmap_and_config 2024-03-29T10:42:07.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.638+0000 7f4db60c1640 10 monclient: build_initial_monmap 2024-03-29T10:42:07.622 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.638+0000 7f4db60c1640 10 monclient: monmap: 2024-03-29T10:42:07.622 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:42:07.622 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:07.622 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:42:07.639499+0000 2024-03-29T10:42:07.622 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:42:07.639499+0000 2024-03-29T10:42:07.622 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:07.622 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:07.622 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:42:07.622 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:42:07.623 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:42:07.623 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:07.623 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.639+0000 7f4db60c1640 1 Processor -- start 2024-03-29T10:42:07.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.639+0000 7f4db60c1640 1 -- start start 2024-03-29T10:42:07.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.639+0000 7f4db60c1640 10 monclient: init 2024-03-29T10:42:07.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.640+0000 7f4db60c1640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:07.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.640+0000 7f4db60c1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:42:07.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.640+0000 7f4db60c1640 10 monclient(hunting): picked mon.noname-b con 0x7f4db00a9750 addr v1:172.21.15.84:6790/0 2024-03-29T10:42:07.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.640+0000 7f4db60c1640 10 monclient(hunting): picked mon.noname-a con 0x7f4db00add40 addr v1:172.21.15.84:6789/0 2024-03-29T10:42:07.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.640+0000 7f4db60c1640 10 monclient(hunting): picked mon.noname-c con 0x7f4db00aad20 addr v1:172.21.15.86:6789/0 2024-03-29T10:42:07.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.640+0000 7f4db60c1640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db0149870 con 0x7f4db00add40 2024-03-29T10:42:07.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.640+0000 7f4db60c1640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db00abf80 con 0x7f4db00aad20 2024-03-29T10:42:07.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.640+0000 7f4db60c1640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db0149ac0 con 0x7f4db00a9750 2024-03-29T10:42:07.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.640+0000 7f4db60c1640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:07.624 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.640+0000 7f4db60c1640 10 monclient(hunting): authenticate will time out at 1229.903777s 2024-03-29T10:42:07.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.643+0000 7f4db50bf640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f4db00a9750 0x7f4db0149fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:46132/0 (socket says 172.21.15.84:46132) 2024-03-29T10:42:07.626 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.643+0000 7f4db50bf640 1 -- 172.21.15.84:0/3235870337 learned_addr learned my addr 172.21.15.84:0/3235870337 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:42:07.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.644+0000 7f4da7fff640 1 -- 172.21.15.84:0/3235870337 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719333541 0 0) 0x7f4db00abf80 con 0x7f4db00aad20 2024-03-29T10:42:07.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.644+0000 7f4da7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:42:07.630 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.644+0000 7f4da7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:07.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.644+0000 7f4da7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:07.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.644+0000 7f4da7fff640 10 monclient(hunting): my global_id is 4444 2024-03-29T10:42:07.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.644+0000 7f4da7fff640 1 -- 172.21.15.84:0/3235870337 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d9c003520 con 0x7f4db00aad20 2024-03-29T10:42:07.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.644+0000 7f4da7fff640 1 -- 172.21.15.84:0/3235870337 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 263690735 0 0) 0x7f4db0149ac0 con 0x7f4db00a9750 2024-03-29T10:42:07.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.644+0000 7f4da7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:42:07.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.644+0000 7f4da7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:07.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.644+0000 7f4da7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:07.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.644+0000 7f4da7fff640 10 monclient(hunting): my global_id is 4541 2024-03-29T10:42:07.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.644+0000 7f4da7fff640 1 -- 172.21.15.84:0/3235870337 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4db00abf80 con 0x7f4db00a9750 2024-03-29T10:42:07.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.645+0000 7f4da7fff640 1 -- 172.21.15.84:0/3235870337 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1521308355 0 0) 0x7f4d9c003520 con 0x7f4db00aad20 2024-03-29T10:42:07.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.645+0000 7f4da7fff640 1 -- 172.21.15.84:0/3235870337 >> v1:172.21.15.84:6790/0 conn(0x7f4db00a9750 legacy=0x7f4db0149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:07.631 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.645+0000 7f4da7fff640 1 -- 172.21.15.84:0/3235870337 >> v1:172.21.15.84:6789/0 conn(0x7f4db00add40 legacy=0x7f4db0143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:42:07.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.645+0000 7f4da7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:42:07.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.645+0000 7f4da7fff640 1 monclient: found mon.noname-c 2024-03-29T10:42:07.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.645+0000 7f4da7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:07.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.645+0000 7f4da7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-29T10:42:07.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.645+0000 7f4da7fff640 1 -- 172.21.15.84:0/3235870337 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db014a6f0 con 0x7f4db00aad20 2024-03-29T10:42:07.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.645+0000 7f4da7fff640 10 monclient: _finish_auth 0 2024-03-29T10:42:07.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.645+0000 7f4da7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:42:07.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.645+0000 7f4da7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:07.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.645+0000 7f4db60c1640 5 monclient: authenticate success, global_id 4444 2024-03-29T10:42:07.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.645+0000 7f4db60c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:07.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.645+0000 7f4da7fff640 1 -- 172.21.15.84:0/3235870337 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4da8001a10 con 0x7f4db00aad20 2024-03-29T10:42:07.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.645+0000 7f4da7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:07.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.645+0000 7f4da7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:07.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.645+0000 7f4da7fff640 10 monclient: dump: 2024-03-29T10:42:07.633 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:07.633 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:07.633 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:07.633 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:07.633 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:07.633 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:07.633 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:07.633 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:07.633 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:07.633 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:07.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.645+0000 7f4db60c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:07.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.647+0000 7f4da7fff640 1 -- 172.21.15.84:0/3235870337 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4da8001c30 con 0x7f4db00aad20 2024-03-29T10:42:07.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.647+0000 7f4da7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:07.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.647+0000 7f4da7fff640 1 -- 172.21.15.84:0/3235870337 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4da8001fc0 con 0x7f4db00aad20 2024-03-29T10:42:07.633 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.647+0000 7f4da7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:07.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.647+0000 7f4da7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:07.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.647+0000 7f4da7fff640 10 monclient: dump: 2024-03-29T10:42:07.634 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:07.634 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:07.634 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:07.634 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:07.634 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:07.634 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:07.634 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:07.634 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:07.634 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:07.634 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:07.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.647+0000 7f4db60c1640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:07.634 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.647+0000 7f4db60c1640 4 set_mon_vals no callback set 2024-03-29T10:42:07.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.648+0000 7f4db60c1640 10 monclient: shutdown 2024-03-29T10:42:07.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.648+0000 7f4db60c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:07.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.648+0000 7f4db60c1640 1 -- 172.21.15.84:0/3235870337 >> v1:172.21.15.86:6789/0 conn(0x7f4db00aad20 legacy=0x7f4db0151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:07.638 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.648+0000 7f4db60c1640 1 -- 172.21.15.84:0/3235870337 shutdown_connections 2024-03-29T10:42:07.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.648+0000 7f4db60c1640 1 -- 172.21.15.84:0/3235870337 >> 172.21.15.84:0/3235870337 conn(0x7f4db00673e0 msgr2=0x7f4db00b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:07.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.648+0000 7f4db60c1640 1 -- 172.21.15.84:0/3235870337 shutdown_connections 2024-03-29T10:42:07.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.649+0000 7f4db60c1640 1 -- 172.21.15.84:0/3235870337 wait complete. 2024-03-29T10:42:07.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.649+0000 7f4db60c1640 10 monclient: build_initial_monmap 2024-03-29T10:42:07.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.649+0000 7f4db60c1640 10 monclient: monmap: 2024-03-29T10:42:07.639 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:42:07.639 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:07.639 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:42:07.639 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:42:07.639 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:07.639 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:07.639 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:42:07.639 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:42:07.639 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:42:07.640 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:07.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.650+0000 7f4db60c1640 1 librados: starting msgr at 2024-03-29T10:42:07.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.650+0000 7f4db60c1640 1 librados: starting objecter 2024-03-29T10:42:07.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.650+0000 7f4db60c1640 1 Processor -- start 2024-03-29T10:42:07.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.650+0000 7f4db60c1640 1 -- start start 2024-03-29T10:42:07.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.650+0000 7f4db60c1640 1 librados: setting wanted keys 2024-03-29T10:42:07.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.650+0000 7f4db60c1640 1 librados: calling monclient init 2024-03-29T10:42:07.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.650+0000 7f4db60c1640 10 monclient: init 2024-03-29T10:42:07.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.651+0000 7f4db60c1640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:07.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.651+0000 7f4db60c1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:42:07.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.651+0000 7f4db60c1640 10 monclient(hunting): picked mon.noname-a con 0x7f4db00a9750 addr v1:172.21.15.84:6789/0 2024-03-29T10:42:07.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.651+0000 7f4db60c1640 10 monclient(hunting): picked mon.noname-c con 0x7f4db00aad20 addr v1:172.21.15.84:6790/0 2024-03-29T10:42:07.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.651+0000 7f4db60c1640 10 monclient(hunting): picked mon.noname-b con 0x7f4db00add40 addr v1:172.21.15.86:6789/0 2024-03-29T10:42:07.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.651+0000 7f4db60c1640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db01e79f0 con 0x7f4db00a9750 2024-03-29T10:42:07.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.651+0000 7f4db60c1640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db01e74d0 con 0x7f4db00add40 2024-03-29T10:42:07.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.651+0000 7f4db60c1640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db01e7660 con 0x7f4db00aad20 2024-03-29T10:42:07.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.651+0000 7f4db60c1640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:07.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.651+0000 7f4db60c1640 10 monclient(hunting): authenticate will time out at 1229.914714s 2024-03-29T10:42:07.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.651+0000 7f4db48be640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f4db00aad20 0x7f4db0146dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:47576/0 (socket says 172.21.15.84:47576) 2024-03-29T10:42:07.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.651+0000 7f4db48be640 1 -- 172.21.15.84:0/3045852610 learned_addr learned my addr 172.21.15.84:0/3045852610 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:42:07.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.652+0000 7f4da5ffb640 10 client.?.objecter ms_handle_connect 0x7f4db00aad20 2024-03-29T10:42:07.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.652+0000 7f4da5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:07.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.652+0000 7f4da5ffb640 1 -- 172.21.15.84:0/3045852610 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 263521897 0 0) 0x7f4db01e7660 con 0x7f4db00aad20 2024-03-29T10:42:07.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.652+0000 7f4da5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:42:07.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.652+0000 7f4da5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:07.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.653+0000 7f4da5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:07.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.653+0000 7f4da5ffb640 10 monclient(hunting): my global_id is 4544 2024-03-29T10:42:07.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.653+0000 7f4da5ffb640 1 -- 172.21.15.84:0/3045852610 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d94004eb0 con 0x7f4db00aad20 2024-03-29T10:42:07.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.653+0000 7f4da5ffb640 1 -- 172.21.15.84:0/3045852610 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 224940410 0 0) 0x7f4d94004eb0 con 0x7f4db00aad20 2024-03-29T10:42:07.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.654+0000 7f4da5ffb640 1 -- 172.21.15.84:0/3045852610 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4db01e7660 con 0x7f4db00aad20 2024-03-29T10:42:07.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.654+0000 7f4da5ffb640 1 -- 172.21.15.84:0/3045852610 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4da0003f80 con 0x7f4db00aad20 2024-03-29T10:42:07.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.654+0000 7f4da5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:07.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.654+0000 7f4da5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:07.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.654+0000 7f4da5ffb640 10 monclient(hunting): dump: 2024-03-29T10:42:07.642 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:07.642 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:07.642 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:07.642 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:07.642 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:07.642 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:07.642 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:07.643 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:07.643 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:07.643 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:07.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.654+0000 7f4da5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:07.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.654+0000 7f4da5ffb640 1 -- 172.21.15.84:0/3045852610 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2107142905 0 0) 0x7f4db01e7660 con 0x7f4db00aad20 2024-03-29T10:42:07.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.654+0000 7f4da5ffb640 1 -- 172.21.15.84:0/3045852610 >> v1:172.21.15.86:6789/0 conn(0x7f4db00add40 legacy=0x7f4db0144170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:42:07.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.654+0000 7f4da5ffb640 1 -- 172.21.15.84:0/3045852610 >> v1:172.21.15.84:6789/0 conn(0x7f4db00a9750 legacy=0x7f4db01e7f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:42:07.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.658+0000 7f4da5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:42:07.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.658+0000 7f4da5ffb640 1 monclient: found mon.c 2024-03-29T10:42:07.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.658+0000 7f4da5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:07.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.658+0000 7f4da5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:42:07.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.658+0000 7f4da5ffb640 1 -- 172.21.15.84:0/3045852610 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db01464d0 con 0x7f4db00aad20 2024-03-29T10:42:07.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.658+0000 7f4da5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:42:07.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.658+0000 7f4da5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:42:07.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.658+0000 7f4da5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:07.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.659+0000 7f4db60c1640 5 monclient: authenticate success, global_id 4544 2024-03-29T10:42:07.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.659+0000 7f4db60c1640 10 monclient: _renew_subs 2024-03-29T10:42:07.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.659+0000 7f4db60c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:42:07.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.659+0000 7f4db60c1640 1 -- 172.21.15.84:0/3045852610 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db01466c0 con 0x7f4db00aad20 2024-03-29T10:42:07.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.659+0000 7f4da5ffb640 1 -- 172.21.15.84:0/3045852610 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4da0004150 con 0x7f4db00aad20 2024-03-29T10:42:07.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.659+0000 7f4db60c1640 10 client.4544.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:07.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.659+0000 7f4db60c1640 10 monclient: _renew_subs 2024-03-29T10:42:07.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.659+0000 7f4db60c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:42:07.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.659+0000 7f4db60c1640 1 -- 172.21.15.84:0/3045852610 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db01d9d40 con 0x7f4db00aad20 2024-03-29T10:42:07.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.659+0000 7f4db60c1640 1 librados: init done 2024-03-29T10:42:07.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.660+0000 7f4da5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:07.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.660+0000 7f4da5ffb640 1 -- 172.21.15.84:0/3045852610 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4da0001130 con 0x7f4db00aad20 2024-03-29T10:42:07.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.660+0000 7f4da5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:07.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.660+0000 7f4da5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:07.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.660+0000 7f4da77fe640 4 set_mon_vals no callback set 2024-03-29T10:42:07.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.660+0000 7f4da5ffb640 10 monclient: dump: 2024-03-29T10:42:07.647 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:07.647 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:07.648 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:07.648 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:07.648 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:07.648 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:07.648 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:07.648 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:07.648 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:07.648 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:07.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.661+0000 7f4da5ffb640 1 -- 172.21.15.84:0/3045852610 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f4da0001290 con 0x7f4db00aad20 2024-03-29T10:42:07.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.662+0000 7f4db60c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:07.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.662+0000 7f4db60c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:07.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.662+0000 7f4db60c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:42:07.649 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.662+0000 7f4db60c1640 1 -- 172.21.15.84:0/3045852610 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db01e77f0 con 0x7f4db00aad20 2024-03-29T10:42:07.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.663+0000 7f4da5ffb640 1 -- 172.21.15.84:0/3045852610 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f4da0084a80 con 0x7f4db00aad20 2024-03-29T10:42:07.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.663+0000 7f4da5ffb640 10 client.4544.objecter ms_dispatch 0x7f4db0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:07.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.664+0000 7f4da5ffb640 3 client.4544.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:07.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.664+0000 7f4da5ffb640 3 client.4544.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:07.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.664+0000 7f4da5ffb640 20 client.4544.objecter dump_active .. 0 homeless 2024-03-29T10:42:07.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.665+0000 7f4db50bf640 10 monclient: get_auth_request con 0x7f4d9406dd60 auth_method 0 2024-03-29T10:42:07.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.665+0000 7f4da5ffb640 10 client.4544.objecter ms_handle_connect 0x7f4d9406dd60 2024-03-29T10:42:07.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.672+0000 7f4da5ffb640 1 -- 172.21.15.84:0/3045852610 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4da00501b0 con 0x7f4db00aad20 2024-03-29T10:42:07.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.672+0000 7f4da5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:07.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.672+0000 7f4da5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:07.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.993+0000 7f4db60c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:07.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.994+0000 7f4db60c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:07.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.994+0000 7f4db60c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:42:07.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.994+0000 7f4db60c1640 1 -- 172.21.15.84:0/3045852610 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4db004f730 con 0x7f4db00aad20 2024-03-29T10:42:07.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.996+0000 7f4da5ffb640 1 -- 172.21.15.84:0/3045852610 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f4da00522b0 con 0x7f4db00aad20 2024-03-29T10:42:07.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.996+0000 7f4da5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:07.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.996+0000 7f4da5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:07.980 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:42:07.980 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:07.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.999+0000 7f4db60c1640 10 librados: watch_flush enter 2024-03-29T10:42:07.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.999+0000 7f4db60c1640 10 librados: watch_flush exit 2024-03-29T10:42:07.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.999+0000 7f4db60c1640 20 client.4544.objecter shutdown clearing up homeless session... 2024-03-29T10:42:07.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:07.999+0000 7f4db60c1640 10 client.4544.objecter successfully canceled tick 2024-03-29T10:42:07.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:08.000+0000 7f4db60c1640 1 -- 172.21.15.84:0/3045852610 >> v1:172.21.15.86:6816/3337225337 conn(0x7f4d9406dd60 legacy=0x7f4d94070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:07.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:08.000+0000 7f4db60c1640 10 monclient: shutdown 2024-03-29T10:42:07.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:08.000+0000 7f4db60c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:07.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:08.000+0000 7f4db60c1640 1 -- 172.21.15.84:0/3045852610 >> v1:172.21.15.84:6790/0 conn(0x7f4db00aad20 legacy=0x7f4db0146dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:07.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:08.002+0000 7f4db60c1640 1 -- 172.21.15.84:0/3045852610 shutdown_connections 2024-03-29T10:42:07.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:08.002+0000 7f4db60c1640 1 -- 172.21.15.84:0/3045852610 >> 172.21.15.84:0/3045852610 conn(0x7f4db00673e0 msgr2=0x7f4db0141cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:07.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:08.002+0000 7f4db60c1640 1 -- 172.21.15.84:0/3045852610 shutdown_connections 2024-03-29T10:42:07.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:08.002+0000 7f4db60c1640 1 -- 172.21.15.84:0/3045852610 wait complete. 2024-03-29T10:42:07.986 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:08.002+0000 7f4db60c1640 1 librados: shutdown 2024-03-29T10:42:08.003 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:14.810 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:14.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:14.992+0000 7f3e7d2f1640 10 monclient: get_monmap_and_config 2024-03-29T10:42:14.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:14.992+0000 7f3e7d2f1640 10 monclient: build_initial_monmap 2024-03-29T10:42:14.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:14.992+0000 7f3e7d2f1640 10 monclient: monmap: 2024-03-29T10:42:14.976 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:42:14.976 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:14.976 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:42:14.993616+0000 2024-03-29T10:42:14.976 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:42:14.993616+0000 2024-03-29T10:42:14.976 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:14.976 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:14.976 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:42:14.976 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:42:14.977 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:42:14.977 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:14.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:14.994+0000 7f3e7d2f1640 1 Processor -- start 2024-03-29T10:42:14.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:14.994+0000 7f3e7d2f1640 1 -- start start 2024-03-29T10:42:14.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:14.994+0000 7f3e7d2f1640 10 monclient: init 2024-03-29T10:42:14.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:14.995+0000 7f3e7d2f1640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:14.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:14.995+0000 7f3e7d2f1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:42:14.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:14.995+0000 7f3e7d2f1640 10 monclient(hunting): picked mon.noname-c con 0x7f3e780b1f60 addr v1:172.21.15.86:6789/0 2024-03-29T10:42:14.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:14.995+0000 7f3e7d2f1640 10 monclient(hunting): picked mon.noname-b con 0x7f3e78071eb0 addr v1:172.21.15.84:6790/0 2024-03-29T10:42:14.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:14.995+0000 7f3e7d2f1640 10 monclient(hunting): picked mon.noname-a con 0x7f3e78143d10 addr v1:172.21.15.84:6789/0 2024-03-29T10:42:14.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:14.995+0000 7f3e7d2f1640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e7806ef80 con 0x7f3e78143d10 2024-03-29T10:42:14.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:14.995+0000 7f3e7d2f1640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e7806f260 con 0x7f3e780b1f60 2024-03-29T10:42:14.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:14.995+0000 7f3e7d2f1640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e780b40c0 con 0x7f3e78071eb0 2024-03-29T10:42:14.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:14.995+0000 7f3e7d2f1640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:14.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:14.995+0000 7f3e7d2f1640 10 monclient(hunting): authenticate will time out at 1237.258815s 2024-03-29T10:42:14.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:14.997+0000 7f3e767fc640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f3e78071eb0 0x7f3e78146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:56006/0 (socket says 172.21.15.84:56006) 2024-03-29T10:42:14.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:14.997+0000 7f3e767fc640 1 -- 172.21.15.84:0/1228695691 learned_addr learned my addr 172.21.15.84:0/1228695691 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:42:14.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:14.999+0000 7f3e75ffb640 1 -- 172.21.15.84:0/1228695691 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2516485058 0 0) 0x7f3e780b40c0 con 0x7f3e78071eb0 2024-03-29T10:42:14.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:14.999+0000 7f3e75ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:42:14.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:14.999+0000 7f3e75ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:14.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:14.999+0000 7f3e75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:14.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:14.999+0000 7f3e75ffb640 10 monclient(hunting): my global_id is 4547 2024-03-29T10:42:14.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:14.999+0000 7f3e75ffb640 1 -- 172.21.15.84:0/1228695691 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e60003520 con 0x7f3e78071eb0 2024-03-29T10:42:14.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.001+0000 7f3e75ffb640 1 -- 172.21.15.84:0/1228695691 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1207691004 0 0) 0x7f3e60003520 con 0x7f3e78071eb0 2024-03-29T10:42:14.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.001+0000 7f3e75ffb640 1 -- 172.21.15.84:0/1228695691 >> v1:172.21.15.86:6789/0 conn(0x7f3e780b1f60 legacy=0x7f3e78143600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:14.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.005+0000 7f3e75ffb640 1 -- 172.21.15.84:0/1228695691 >> v1:172.21.15.84:6789/0 conn(0x7f3e78143d10 legacy=0x7f3e78146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:14.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.005+0000 7f3e75ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:42:14.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.005+0000 7f3e75ffb640 1 monclient: found mon.noname-b 2024-03-29T10:42:14.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.005+0000 7f3e75ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:14.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.005+0000 7f3e75ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:42:14.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.005+0000 7f3e75ffb640 1 -- 172.21.15.84:0/1228695691 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e78155d70 con 0x7f3e78071eb0 2024-03-29T10:42:14.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.005+0000 7f3e75ffb640 10 monclient: _finish_auth 0 2024-03-29T10:42:14.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.005+0000 7f3e75ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:42:14.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.005+0000 7f3e75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:14.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.005+0000 7f3e75ffb640 1 -- 172.21.15.84:0/1228695691 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f3e6c001690 con 0x7f3e78071eb0 2024-03-29T10:42:14.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.005+0000 7f3e75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:14.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.005+0000 7f3e75ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:14.992 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.005+0000 7f3e75ffb640 10 monclient: dump: 2024-03-29T10:42:14.992 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:14.992 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:14.992 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:14.992 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:14.992 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:14.993 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:14.993 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:14.993 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:14.993 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:14.993 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:14.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.006+0000 7f3e7d2f1640 5 monclient: authenticate success, global_id 4547 2024-03-29T10:42:14.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.006+0000 7f3e7d2f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:14.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.006+0000 7f3e75ffb640 1 -- 172.21.15.84:0/1228695691 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e6c001920 con 0x7f3e78071eb0 2024-03-29T10:42:14.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.006+0000 7f3e75ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:14.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.006+0000 7f3e7d2f1640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:14.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.006+0000 7f3e7d2f1640 4 set_mon_vals no callback set 2024-03-29T10:42:14.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.007+0000 7f3e7d2f1640 10 monclient: shutdown 2024-03-29T10:42:14.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.007+0000 7f3e7d2f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:14.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.007+0000 7f3e7d2f1640 1 -- 172.21.15.84:0/1228695691 >> v1:172.21.15.84:6790/0 conn(0x7f3e78071eb0 legacy=0x7f3e78146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:14.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.007+0000 7f3e7d2f1640 1 -- 172.21.15.84:0/1228695691 shutdown_connections 2024-03-29T10:42:14.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.007+0000 7f3e7d2f1640 1 -- 172.21.15.84:0/1228695691 >> 172.21.15.84:0/1228695691 conn(0x7f3e78066a00 msgr2=0x7f3e78066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:14.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.008+0000 7f3e7d2f1640 1 -- 172.21.15.84:0/1228695691 shutdown_connections 2024-03-29T10:42:14.997 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.008+0000 7f3e7d2f1640 1 -- 172.21.15.84:0/1228695691 wait complete. 2024-03-29T10:42:14.997 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.009+0000 7f3e7d2f1640 10 monclient: build_initial_monmap 2024-03-29T10:42:14.997 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.009+0000 7f3e7d2f1640 10 monclient: monmap: 2024-03-29T10:42:14.997 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:42:14.997 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:14.997 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:42:14.997 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:42:14.997 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:14.997 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:14.997 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:42:14.997 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:42:14.997 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:42:14.998 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:14.998 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.009+0000 7f3e7d2f1640 1 librados: starting msgr at 2024-03-29T10:42:14.998 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.009+0000 7f3e7d2f1640 1 librados: starting objecter 2024-03-29T10:42:14.998 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.009+0000 7f3e7d2f1640 1 Processor -- start 2024-03-29T10:42:14.998 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.010+0000 7f3e7d2f1640 1 -- start start 2024-03-29T10:42:14.998 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.010+0000 7f3e7d2f1640 1 librados: setting wanted keys 2024-03-29T10:42:14.998 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.010+0000 7f3e7d2f1640 1 librados: calling monclient init 2024-03-29T10:42:14.998 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.010+0000 7f3e7d2f1640 10 monclient: init 2024-03-29T10:42:14.998 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.011+0000 7f3e7d2f1640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:14.998 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.011+0000 7f3e7d2f1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:42:14.998 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.011+0000 7f3e7d2f1640 10 monclient(hunting): picked mon.noname-c con 0x7f3e780b1f60 addr v1:172.21.15.84:6790/0 2024-03-29T10:42:14.998 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.011+0000 7f3e7d2f1640 10 monclient(hunting): picked mon.noname-a con 0x7f3e78143d10 addr v1:172.21.15.84:6789/0 2024-03-29T10:42:14.998 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.011+0000 7f3e7d2f1640 10 monclient(hunting): picked mon.noname-b con 0x7f3e781379e0 addr v1:172.21.15.86:6789/0 2024-03-29T10:42:14.998 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.011+0000 7f3e7d2f1640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e780b7830 con 0x7f3e78143d10 2024-03-29T10:42:14.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.011+0000 7f3e7d2f1640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e780b7af0 con 0x7f3e781379e0 2024-03-29T10:42:14.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.011+0000 7f3e7d2f1640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e780b8a30 con 0x7f3e780b1f60 2024-03-29T10:42:14.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.011+0000 7f3e7d2f1640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:14.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.011+0000 7f3e7d2f1640 10 monclient(hunting): authenticate will time out at 1237.274606s 2024-03-29T10:42:14.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.011+0000 7f3e76ffd640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f3e780b1f60 0x7f3e780b7120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:59254/0 (socket says 172.21.15.84:59254) 2024-03-29T10:42:14.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.011+0000 7f3e76ffd640 1 -- 172.21.15.84:0/1454396389 learned_addr learned my addr 172.21.15.84:0/1454396389 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:42:14.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.013+0000 7f3e5f7fe640 10 client.?.objecter ms_handle_connect 0x7f3e780b1f60 2024-03-29T10:42:14.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.013+0000 7f3e5f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:14.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.013+0000 7f3e5f7fe640 1 -- 172.21.15.84:0/1454396389 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1408062059 0 0) 0x7f3e780b8a30 con 0x7f3e780b1f60 2024-03-29T10:42:14.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.013+0000 7f3e5f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:42:14.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.013+0000 7f3e5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:14.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.013+0000 7f3e5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:14.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.013+0000 7f3e5f7fe640 10 monclient(hunting): my global_id is 4550 2024-03-29T10:42:14.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.013+0000 7f3e5f7fe640 1 -- 172.21.15.84:0/1454396389 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e64004eb0 con 0x7f3e780b1f60 2024-03-29T10:42:15.002 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.014+0000 7f3e5f7fe640 1 -- 172.21.15.84:0/1454396389 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 857521701 0 0) 0x7f3e64004eb0 con 0x7f3e780b1f60 2024-03-29T10:42:15.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.015+0000 7f3e5f7fe640 1 -- 172.21.15.84:0/1454396389 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e780b8a30 con 0x7f3e780b1f60 2024-03-29T10:42:15.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.015+0000 7f3e5f7fe640 1 -- 172.21.15.84:0/1454396389 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f3e70004400 con 0x7f3e780b1f60 2024-03-29T10:42:15.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.015+0000 7f3e5f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:15.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.015+0000 7f3e5f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:15.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.015+0000 7f3e5f7fe640 10 monclient(hunting): dump: 2024-03-29T10:42:15.003 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:15.003 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:15.003 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:15.003 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:15.003 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:15.003 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:15.003 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:15.003 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:15.004 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:15.004 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:15.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.015+0000 7f3e5f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:15.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.015+0000 7f3e5f7fe640 1 -- 172.21.15.84:0/1454396389 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3329528764 0 0) 0x7f3e780b8a30 con 0x7f3e780b1f60 2024-03-29T10:42:15.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.015+0000 7f3e5f7fe640 1 -- 172.21.15.84:0/1454396389 >> v1:172.21.15.86:6789/0 conn(0x7f3e781379e0 legacy=0x7f3e781e5ea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:42:15.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.016+0000 7f3e5f7fe640 1 -- 172.21.15.84:0/1454396389 >> v1:172.21.15.84:6789/0 conn(0x7f3e78143d10 legacy=0x7f3e781372d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:15.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.016+0000 7f3e5f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:42:15.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.016+0000 7f3e5f7fe640 1 monclient: found mon.c 2024-03-29T10:42:15.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.016+0000 7f3e5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:15.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.016+0000 7f3e5f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:42:15.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.016+0000 7f3e5f7fe640 1 -- 172.21.15.84:0/1454396389 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e78138f90 con 0x7f3e780b1f60 2024-03-29T10:42:15.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.016+0000 7f3e5f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:42:15.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.016+0000 7f3e5f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:42:15.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.016+0000 7f3e5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:15.004 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.016+0000 7f3e5f7fe640 1 -- 172.21.15.84:0/1454396389 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3e70004660 con 0x7f3e780b1f60 2024-03-29T10:42:15.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.016+0000 7f3e5f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:15.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.017+0000 7f3e7d2f1640 5 monclient: authenticate success, global_id 4550 2024-03-29T10:42:15.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.017+0000 7f3e7d2f1640 10 monclient: _renew_subs 2024-03-29T10:42:15.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.017+0000 7f3e7d2f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:42:15.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.017+0000 7f3e7d2f1640 1 -- 172.21.15.84:0/1454396389 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e78137e90 con 0x7f3e780b1f60 2024-03-29T10:42:15.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.017+0000 7f3e7d2f1640 10 client.4550.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:15.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.017+0000 7f3e7d2f1640 10 monclient: _renew_subs 2024-03-29T10:42:15.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.017+0000 7f3e7d2f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:42:15.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.017+0000 7f3e7d2f1640 1 -- 172.21.15.84:0/1454396389 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e780a94e0 con 0x7f3e780b1f60 2024-03-29T10:42:15.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.017+0000 7f3e7d2f1640 1 librados: init done 2024-03-29T10:42:15.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.017+0000 7f3e74ff9640 4 set_mon_vals no callback set 2024-03-29T10:42:15.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.019+0000 7f3e5f7fe640 1 -- 172.21.15.84:0/1454396389 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f3e700016a0 con 0x7f3e780b1f60 2024-03-29T10:42:15.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.019+0000 7f3e5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:15.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.019+0000 7f3e5f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:15.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.019+0000 7f3e5f7fe640 10 monclient: dump: 2024-03-29T10:42:15.007 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:15.007 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:15.007 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:15.007 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:15.007 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:15.007 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:15.007 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:15.007 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:15.007 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:15.007 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:15.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.020+0000 7f3e5f7fe640 1 -- 172.21.15.84:0/1454396389 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f3e70001900 con 0x7f3e780b1f60 2024-03-29T10:42:15.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.020+0000 7f3e7d2f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:15.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.020+0000 7f3e7d2f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:15.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.020+0000 7f3e7d2f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:42:15.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.020+0000 7f3e7d2f1640 1 -- 172.21.15.84:0/1454396389 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e78075f10 con 0x7f3e780b1f60 2024-03-29T10:42:15.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.020+0000 7f3e5f7fe640 1 -- 172.21.15.84:0/1454396389 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f3e70084fe0 con 0x7f3e780b1f60 2024-03-29T10:42:15.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.020+0000 7f3e5f7fe640 10 client.4550.objecter ms_dispatch 0x7f3e78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:15.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.020+0000 7f3e5f7fe640 3 client.4550.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:15.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.020+0000 7f3e5f7fe640 3 client.4550.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:15.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.020+0000 7f3e5f7fe640 20 client.4550.objecter dump_active .. 0 homeless 2024-03-29T10:42:15.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.021+0000 7f3e767fc640 10 monclient: get_auth_request con 0x7f3e6406db30 auth_method 0 2024-03-29T10:42:15.008 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.022+0000 7f3e5f7fe640 10 client.4550.objecter ms_handle_connect 0x7f3e6406db30 2024-03-29T10:42:15.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.032+0000 7f3e5f7fe640 1 -- 172.21.15.84:0/1454396389 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3e70050970 con 0x7f3e780b1f60 2024-03-29T10:42:15.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.032+0000 7f3e5f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:15.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.032+0000 7f3e5f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:15.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.386+0000 7f3e7d2f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:15.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.387+0000 7f3e7d2f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:15.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.387+0000 7f3e7d2f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:42:15.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.387+0000 7f3e7d2f1640 1 -- 172.21.15.84:0/1454396389 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e78152af0 con 0x7f3e780b1f60 2024-03-29T10:42:15.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.388+0000 7f3e5f7fe640 1 -- 172.21.15.84:0/1454396389 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f3e700528a0 con 0x7f3e780b1f60 2024-03-29T10:42:15.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.388+0000 7f3e5f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:15.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.388+0000 7f3e5f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:15.372 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:42:15.372 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:15.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.392+0000 7f3e7d2f1640 10 librados: watch_flush enter 2024-03-29T10:42:15.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.392+0000 7f3e7d2f1640 10 librados: watch_flush exit 2024-03-29T10:42:15.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.392+0000 7f3e7d2f1640 20 client.4550.objecter shutdown clearing up homeless session... 2024-03-29T10:42:15.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.392+0000 7f3e7d2f1640 10 client.4550.objecter successfully canceled tick 2024-03-29T10:42:15.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.392+0000 7f3e7d2f1640 1 -- 172.21.15.84:0/1454396389 >> v1:172.21.15.86:6816/3337225337 conn(0x7f3e6406db30 legacy=0x7f3e6406fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:15.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.392+0000 7f3e7d2f1640 10 monclient: shutdown 2024-03-29T10:42:15.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.392+0000 7f3e7d2f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:15.376 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.392+0000 7f3e7d2f1640 1 -- 172.21.15.84:0/1454396389 >> v1:172.21.15.84:6790/0 conn(0x7f3e780b1f60 legacy=0x7f3e780b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:15.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.392+0000 7f3e7d2f1640 1 -- 172.21.15.84:0/1454396389 shutdown_connections 2024-03-29T10:42:15.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.392+0000 7f3e7d2f1640 1 -- 172.21.15.84:0/1454396389 >> 172.21.15.84:0/1454396389 conn(0x7f3e78066a00 msgr2=0x7f3e78066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:15.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.393+0000 7f3e7d2f1640 1 -- 172.21.15.84:0/1454396389 shutdown_connections 2024-03-29T10:42:15.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.393+0000 7f3e7d2f1640 1 -- 172.21.15.84:0/1454396389 wait complete. 2024-03-29T10:42:15.377 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:15.393+0000 7f3e7d2f1640 1 librados: shutdown 2024-03-29T10:42:15.393 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:22.200 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:22.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.350+0000 7f40248e6640 10 monclient: get_monmap_and_config 2024-03-29T10:42:22.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.350+0000 7f40248e6640 10 monclient: build_initial_monmap 2024-03-29T10:42:22.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.350+0000 7f40248e6640 10 monclient: monmap: 2024-03-29T10:42:22.334 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:42:22.334 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:22.334 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:42:22.351313+0000 2024-03-29T10:42:22.334 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:42:22.351313+0000 2024-03-29T10:42:22.334 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:22.334 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:22.334 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:42:22.334 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:42:22.335 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:42:22.335 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:22.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.351+0000 7f40248e6640 1 Processor -- start 2024-03-29T10:42:22.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.351+0000 7f40248e6640 1 -- start start 2024-03-29T10:42:22.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.351+0000 7f40248e6640 10 monclient: init 2024-03-29T10:42:22.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.352+0000 7f40248e6640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:22.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.352+0000 7f40248e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:42:22.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.352+0000 7f40248e6640 10 monclient(hunting): picked mon.noname-c con 0x7f40200a9750 addr v1:172.21.15.86:6789/0 2024-03-29T10:42:22.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.352+0000 7f40248e6640 10 monclient(hunting): picked mon.noname-a con 0x7f402013e1e0 addr v1:172.21.15.84:6789/0 2024-03-29T10:42:22.337 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.352+0000 7f401dd9b640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f402013e1e0 0x7f40200ac710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:47454/0 (socket says 172.21.15.84:47454) 2024-03-29T10:42:22.337 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.352+0000 7f40248e6640 10 monclient(hunting): picked mon.noname-b con 0x7f4020071c30 addr v1:172.21.15.84:6790/0 2024-03-29T10:42:22.337 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.352+0000 7f401dd9b640 1 -- 172.21.15.84:0/3912043779 learned_addr learned my addr 172.21.15.84:0/3912043779 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:42:22.337 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.352+0000 7f40248e6640 1 -- 172.21.15.84:0/3912043779 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40200ab300 con 0x7f402013e1e0 2024-03-29T10:42:22.337 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.352+0000 7f40248e6640 1 -- 172.21.15.84:0/3912043779 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40200ac2f0 con 0x7f40200a9750 2024-03-29T10:42:22.337 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.352+0000 7f40248e6640 1 -- 172.21.15.84:0/3912043779 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40200ad8d0 con 0x7f4020071c30 2024-03-29T10:42:22.337 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.352+0000 7f40248e6640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:22.337 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.352+0000 7f40248e6640 10 monclient(hunting): authenticate will time out at 1244.615844s 2024-03-29T10:42:22.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.353+0000 7f401d59a640 1 -- 172.21.15.84:0/3912043779 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922493987 0 0) 0x7f40200ad8d0 con 0x7f4020071c30 2024-03-29T10:42:22.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.353+0000 7f401d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:42:22.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.353+0000 7f401d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:22.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.353+0000 7f401d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:22.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.353+0000 7f401d59a640 10 monclient(hunting): my global_id is 4553 2024-03-29T10:42:22.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.353+0000 7f401d59a640 1 -- 172.21.15.84:0/3912043779 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4008003520 con 0x7f4020071c30 2024-03-29T10:42:22.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.353+0000 7f401d59a640 1 -- 172.21.15.84:0/3912043779 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3321019070 0 0) 0x7f4008003520 con 0x7f4020071c30 2024-03-29T10:42:22.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.353+0000 7f401d59a640 1 -- 172.21.15.84:0/3912043779 >> v1:172.21.15.86:6789/0 conn(0x7f40200a9750 legacy=0x7f402013dad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:22.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.356+0000 7f401d59a640 1 -- 172.21.15.84:0/3912043779 >> v1:172.21.15.84:6789/0 conn(0x7f402013e1e0 legacy=0x7f40200ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:22.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.356+0000 7f401d59a640 10 monclient: _finish_hunting 0 2024-03-29T10:42:22.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.356+0000 7f401d59a640 1 monclient: found mon.noname-b 2024-03-29T10:42:22.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.356+0000 7f401d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:22.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.356+0000 7f401d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:42:22.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.356+0000 7f401d59a640 1 -- 172.21.15.84:0/3912043779 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40200ada60 con 0x7f4020071c30 2024-03-29T10:42:22.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.356+0000 7f401d59a640 10 monclient: _finish_auth 0 2024-03-29T10:42:22.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.356+0000 7f401d59a640 10 monclient: _check_auth_tickets 2024-03-29T10:42:22.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.356+0000 7f401d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:22.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.356+0000 7f401d59a640 1 -- 172.21.15.84:0/3912043779 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4018001a10 con 0x7f4020071c30 2024-03-29T10:42:22.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.356+0000 7f40248e6640 5 monclient: authenticate success, global_id 4553 2024-03-29T10:42:22.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.356+0000 7f40248e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:22.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.356+0000 7f401d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:22.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.356+0000 7f401d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:22.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.356+0000 7f401d59a640 10 monclient: dump: 2024-03-29T10:42:22.340 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:22.341 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:22.341 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:22.341 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:22.341 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:22.341 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:22.341 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:22.341 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:22.341 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:22.341 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:22.341 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.356+0000 7f40248e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:22.342 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.357+0000 7f401d59a640 1 -- 172.21.15.84:0/3912043779 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4018001be0 con 0x7f4020071c30 2024-03-29T10:42:22.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.357+0000 7f401d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:22.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.357+0000 7f401d59a640 1 -- 172.21.15.84:0/3912043779 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4018001ff0 con 0x7f4020071c30 2024-03-29T10:42:22.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.357+0000 7f40248e6640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:22.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.357+0000 7f40248e6640 4 set_mon_vals no callback set 2024-03-29T10:42:22.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.357+0000 7f401d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:22.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.357+0000 7f401d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:22.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.357+0000 7f401d59a640 10 monclient: dump: 2024-03-29T10:42:22.343 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:22.343 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:22.343 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:22.343 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:22.343 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:22.343 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:22.343 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:22.344 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:22.344 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:22.344 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:22.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.358+0000 7f40248e6640 10 monclient: shutdown 2024-03-29T10:42:22.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.358+0000 7f40248e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:22.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.358+0000 7f40248e6640 1 -- 172.21.15.84:0/3912043779 >> v1:172.21.15.84:6790/0 conn(0x7f4020071c30 legacy=0x7f40200ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:22.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.359+0000 7f40248e6640 1 -- 172.21.15.84:0/3912043779 shutdown_connections 2024-03-29T10:42:22.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.359+0000 7f40248e6640 1 -- 172.21.15.84:0/3912043779 >> 172.21.15.84:0/3912043779 conn(0x7f402006b660 msgr2=0x7f402006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:22.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.359+0000 7f40248e6640 1 -- 172.21.15.84:0/3912043779 shutdown_connections 2024-03-29T10:42:22.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.359+0000 7f40248e6640 1 -- 172.21.15.84:0/3912043779 wait complete. 2024-03-29T10:42:22.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.360+0000 7f40248e6640 10 monclient: build_initial_monmap 2024-03-29T10:42:22.350 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.360+0000 7f40248e6640 10 monclient: monmap: 2024-03-29T10:42:22.351 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:42:22.351 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:22.351 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:42:22.351 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:42:22.351 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:22.351 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:22.351 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:42:22.351 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:42:22.351 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:42:22.351 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:22.351 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.360+0000 7f40248e6640 1 librados: starting msgr at 2024-03-29T10:42:22.351 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.360+0000 7f40248e6640 1 librados: starting objecter 2024-03-29T10:42:22.351 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.360+0000 7f40248e6640 1 Processor -- start 2024-03-29T10:42:22.351 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.360+0000 7f40248e6640 1 -- start start 2024-03-29T10:42:22.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.360+0000 7f40248e6640 1 librados: setting wanted keys 2024-03-29T10:42:22.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.360+0000 7f40248e6640 1 librados: calling monclient init 2024-03-29T10:42:22.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.360+0000 7f40248e6640 10 monclient: init 2024-03-29T10:42:22.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.361+0000 7f40248e6640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:22.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.361+0000 7f40248e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:42:22.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.361+0000 7f40248e6640 10 monclient(hunting): picked mon.noname-c con 0x7f4020071c30 addr v1:172.21.15.84:6790/0 2024-03-29T10:42:22.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.362+0000 7f401e59c640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f4020071c30 0x7f4020133c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:60752/0 (socket says 172.21.15.84:60752) 2024-03-29T10:42:22.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.362+0000 7f401dd9b640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f40200a9750 0x7f4020135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:49558/0 (socket says 172.21.15.84:49558) 2024-03-29T10:42:22.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.362+0000 7f401e59c640 1 -- 172.21.15.84:0/1585466842 learned_addr learned my addr 172.21.15.84:0/1585466842 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:42:22.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.362+0000 7f40248e6640 10 monclient(hunting): picked mon.noname-a con 0x7f40200a9750 addr v1:172.21.15.84:6789/0 2024-03-29T10:42:22.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.362+0000 7f400effd640 10 client.?.objecter ms_handle_connect 0x7f4020071c30 2024-03-29T10:42:22.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.362+0000 7f400effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:22.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.362+0000 7f400effd640 10 client.?.objecter ms_handle_connect 0x7f40200a9750 2024-03-29T10:42:22.352 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.362+0000 7f400effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:22.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.362+0000 7f40248e6640 10 monclient(hunting): picked mon.noname-b con 0x7f402013e1e0 addr v1:172.21.15.86:6789/0 2024-03-29T10:42:22.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.362+0000 7f40248e6640 1 -- 172.21.15.84:0/1585466842 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40201343a0 con 0x7f40200a9750 2024-03-29T10:42:22.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.362+0000 7f40248e6640 1 -- 172.21.15.84:0/1585466842 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4020135540 con 0x7f402013e1e0 2024-03-29T10:42:22.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.362+0000 7f40248e6640 1 -- 172.21.15.84:0/1585466842 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4020136650 con 0x7f4020071c30 2024-03-29T10:42:22.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.362+0000 7f400effd640 1 -- 172.21.15.84:0/1585466842 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3854838954 0 0) 0x7f40201343a0 con 0x7f40200a9750 2024-03-29T10:42:22.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.362+0000 7f40248e6640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:22.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.363+0000 7f40248e6640 10 monclient(hunting): authenticate will time out at 1244.626115s 2024-03-29T10:42:22.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.363+0000 7f400effd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:22.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.363+0000 7f400effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:22.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.363+0000 7f400effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:22.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.363+0000 7f400effd640 10 monclient(hunting): my global_id is 4473 2024-03-29T10:42:22.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.363+0000 7f400effd640 1 -- 172.21.15.84:0/1585466842 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4000004eb0 con 0x7f40200a9750 2024-03-29T10:42:22.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.363+0000 7f400effd640 1 -- 172.21.15.84:0/1585466842 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4221754501 0 0) 0x7f4020136650 con 0x7f4020071c30 2024-03-29T10:42:22.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.363+0000 7f400effd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:22.353 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.363+0000 7f400effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:22.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.363+0000 7f400effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:22.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.363+0000 7f400effd640 10 monclient(hunting): my global_id is 4556 2024-03-29T10:42:22.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.363+0000 7f400effd640 1 -- 172.21.15.84:0/1585466842 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40201343a0 con 0x7f4020071c30 2024-03-29T10:42:22.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.363+0000 7f400effd640 1 -- 172.21.15.84:0/1585466842 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1934805731 0 0) 0x7f4000004eb0 con 0x7f40200a9750 2024-03-29T10:42:22.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.363+0000 7f400effd640 1 -- 172.21.15.84:0/1585466842 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4020136650 con 0x7f40200a9750 2024-03-29T10:42:22.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.363+0000 7f400effd640 1 -- 172.21.15.84:0/1585466842 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4014001780 con 0x7f40200a9750 2024-03-29T10:42:22.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.363+0000 7f400effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:22.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.363+0000 7f400effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:22.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.363+0000 7f400effd640 10 monclient(hunting): dump: 2024-03-29T10:42:22.354 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:22.354 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:22.355 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:22.355 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:22.355 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:22.355 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:22.355 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:22.355 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:22.355 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:22.355 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:22.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.363+0000 7f400effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:22.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.363+0000 7f400effd640 1 -- 172.21.15.84:0/1585466842 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3686234972 0 0) 0x7f40201343a0 con 0x7f4020071c30 2024-03-29T10:42:22.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.363+0000 7f400effd640 1 -- 172.21.15.84:0/1585466842 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4000004eb0 con 0x7f4020071c30 2024-03-29T10:42:22.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.363+0000 7f400effd640 1 -- 172.21.15.84:0/1585466842 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4010001830 con 0x7f4020071c30 2024-03-29T10:42:22.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.363+0000 7f400effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:22.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.363+0000 7f400effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:22.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.363+0000 7f400effd640 10 monclient(hunting): dump: 2024-03-29T10:42:22.356 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:22.356 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:22.356 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:22.356 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:22.356 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:22.356 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:22.356 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:22.356 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:22.356 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:22.356 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:22.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.363+0000 7f400effd640 1 -- 172.21.15.84:0/1585466842 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3375611795 0 0) 0x7f4020136650 con 0x7f40200a9750 2024-03-29T10:42:22.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.363+0000 7f400effd640 1 -- 172.21.15.84:0/1585466842 >> v1:172.21.15.84:6790/0 conn(0x7f4020071c30 legacy=0x7f4020133c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:22.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.363+0000 7f400effd640 1 -- 172.21.15.84:0/1585466842 >> v1:172.21.15.86:6789/0 conn(0x7f402013e1e0 legacy=0x7f40201dd850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:42:22.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.365+0000 7f400effd640 10 monclient: _finish_hunting 0 2024-03-29T10:42:22.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.365+0000 7f400effd640 1 monclient: found mon.a 2024-03-29T10:42:22.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.365+0000 7f400effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:22.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.365+0000 7f400effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:42:22.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.365+0000 7f400effd640 1 -- 172.21.15.84:0/1585466842 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40201357b0 con 0x7f40200a9750 2024-03-29T10:42:22.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.365+0000 7f400effd640 10 monclient: _finish_auth 0 2024-03-29T10:42:22.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.365+0000 7f400effd640 10 monclient: _check_auth_tickets 2024-03-29T10:42:22.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.365+0000 7f400effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:22.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.366+0000 7f40248e6640 5 monclient: authenticate success, global_id 4473 2024-03-29T10:42:22.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.366+0000 7f40248e6640 10 monclient: _renew_subs 2024-03-29T10:42:22.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.366+0000 7f40248e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:42:22.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.366+0000 7f40248e6640 1 -- 172.21.15.84:0/1585466842 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4020142010 con 0x7f40200a9750 2024-03-29T10:42:22.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.366+0000 7f40248e6640 10 client.4473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:22.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.366+0000 7f40248e6640 10 monclient: _renew_subs 2024-03-29T10:42:22.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.366+0000 7f40248e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:42:22.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.366+0000 7f40248e6640 1 -- 172.21.15.84:0/1585466842 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4020133680 con 0x7f40200a9750 2024-03-29T10:42:22.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.366+0000 7f40248e6640 1 librados: init done 2024-03-29T10:42:22.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.366+0000 7f400effd640 1 -- 172.21.15.84:0/1585466842 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4014001c90 con 0x7f40200a9750 2024-03-29T10:42:22.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.366+0000 7f400effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:22.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.366+0000 7f400effd640 1 -- 172.21.15.84:0/1585466842 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4014004980 con 0x7f40200a9750 2024-03-29T10:42:22.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.366+0000 7f400effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:22.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.366+0000 7f400effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:22.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.366+0000 7f400effd640 10 monclient: dump: 2024-03-29T10:42:22.358 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:22.358 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:22.359 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:22.359 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:22.359 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:22.359 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:22.359 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:22.359 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:22.359 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:22.359 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:22.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.366+0000 7f401cd99640 4 set_mon_vals no callback set 2024-03-29T10:42:22.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.368+0000 7f400effd640 1 -- 172.21.15.84:0/1585466842 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f401401a950 con 0x7f40200a9750 2024-03-29T10:42:22.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.369+0000 7f40248e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:22.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.369+0000 7f40248e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:22.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.369+0000 7f40248e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:42:22.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.369+0000 7f40248e6640 1 -- 172.21.15.84:0/1585466842 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4004005080 con 0x7f40200a9750 2024-03-29T10:42:22.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.369+0000 7f400effd640 1 -- 172.21.15.84:0/1585466842 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f40140848e0 con 0x7f40200a9750 2024-03-29T10:42:22.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.369+0000 7f400effd640 10 client.4473.objecter ms_dispatch 0x7f4020000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:22.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.369+0000 7f400effd640 3 client.4473.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:22.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.369+0000 7f400effd640 3 client.4473.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:22.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.369+0000 7f400effd640 20 client.4473.objecter dump_active .. 0 homeless 2024-03-29T10:42:22.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.369+0000 7f401e59c640 10 monclient: get_auth_request con 0x7f400006ddf0 auth_method 0 2024-03-29T10:42:22.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.370+0000 7f400effd640 10 client.4473.objecter ms_handle_connect 0x7f400006ddf0 2024-03-29T10:42:22.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.377+0000 7f400effd640 1 -- 172.21.15.84:0/1585466842 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f40140501c0 con 0x7f40200a9750 2024-03-29T10:42:22.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.377+0000 7f400effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:22.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.377+0000 7f400effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:22.702 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.717+0000 7f40248e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:22.702 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.717+0000 7f40248e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:22.702 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.717+0000 7f40248e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:42:22.702 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.717+0000 7f40248e6640 1 -- 172.21.15.84:0/1585466842 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4004005330 con 0x7f40200a9750 2024-03-29T10:42:22.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.719+0000 7f400effd640 1 -- 172.21.15.84:0/1585466842 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f40140520f0 con 0x7f40200a9750 2024-03-29T10:42:22.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.719+0000 7f400effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:22.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.719+0000 7f400effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:22.703 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:42:22.704 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:22.707 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.723+0000 7f40248e6640 10 librados: watch_flush enter 2024-03-29T10:42:22.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.725+0000 7f40248e6640 10 librados: watch_flush exit 2024-03-29T10:42:22.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.725+0000 7f40248e6640 20 client.4473.objecter shutdown clearing up homeless session... 2024-03-29T10:42:22.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.725+0000 7f40248e6640 10 client.4473.objecter successfully canceled tick 2024-03-29T10:42:22.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.725+0000 7f40248e6640 1 -- 172.21.15.84:0/1585466842 >> v1:172.21.15.86:6816/3337225337 conn(0x7f400006ddf0 legacy=0x7f40000702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:22.709 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.725+0000 7f40248e6640 10 monclient: shutdown 2024-03-29T10:42:22.709 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.725+0000 7f40248e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:22.709 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.725+0000 7f40248e6640 1 -- 172.21.15.84:0/1585466842 >> v1:172.21.15.84:6789/0 conn(0x7f40200a9750 legacy=0x7f4020135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:22.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.729+0000 7f40248e6640 1 -- 172.21.15.84:0/1585466842 shutdown_connections 2024-03-29T10:42:22.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.729+0000 7f40248e6640 1 -- 172.21.15.84:0/1585466842 >> 172.21.15.84:0/1585466842 conn(0x7f402006b660 msgr2=0x7f402013ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:22.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.730+0000 7f40248e6640 1 -- 172.21.15.84:0/1585466842 shutdown_connections 2024-03-29T10:42:22.713 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.730+0000 7f40248e6640 1 -- 172.21.15.84:0/1585466842 wait complete. 2024-03-29T10:42:22.714 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:22.731+0000 7f40248e6640 1 librados: shutdown 2024-03-29T10:42:22.733 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:29.541 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:29.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.700+0000 7f9c9de4a640 10 monclient: get_monmap_and_config 2024-03-29T10:42:29.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.701+0000 7f9c9de4a640 10 monclient: build_initial_monmap 2024-03-29T10:42:29.684 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.701+0000 7f9c9de4a640 10 monclient: monmap: 2024-03-29T10:42:29.684 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:42:29.685 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:29.685 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:42:29.701988+0000 2024-03-29T10:42:29.685 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:42:29.701988+0000 2024-03-29T10:42:29.685 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:29.685 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:29.685 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:42:29.685 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:42:29.685 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:42:29.685 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:29.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.702+0000 7f9c9de4a640 1 Processor -- start 2024-03-29T10:42:29.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.702+0000 7f9c9de4a640 1 -- start start 2024-03-29T10:42:29.686 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.702+0000 7f9c9de4a640 10 monclient: init 2024-03-29T10:42:29.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.703+0000 7f9c9de4a640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:29.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.703+0000 7f9c9de4a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:42:29.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.703+0000 7f9c9de4a640 10 monclient(hunting): picked mon.noname-c con 0x7f9c981457f0 addr v1:172.21.15.86:6789/0 2024-03-29T10:42:29.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.703+0000 7f9c9de4a640 10 monclient(hunting): picked mon.noname-b con 0x7f9c9813f650 addr v1:172.21.15.84:6790/0 2024-03-29T10:42:29.687 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.703+0000 7f9c9de4a640 10 monclient(hunting): picked mon.noname-a con 0x7f9c980a9d70 addr v1:172.21.15.84:6789/0 2024-03-29T10:42:29.688 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.703+0000 7f9c9de4a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c980aaef0 con 0x7f9c980a9d70 2024-03-29T10:42:29.688 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.703+0000 7f9c9de4a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c98145bd0 con 0x7f9c981457f0 2024-03-29T10:42:29.688 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.703+0000 7f9c9de4a640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c98145eb0 con 0x7f9c9813f650 2024-03-29T10:42:29.688 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.703+0000 7f9c9de4a640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:29.688 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.703+0000 7f9c9de4a640 10 monclient(hunting): authenticate will time out at 1251.966309s 2024-03-29T10:42:29.688 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.703+0000 7f9c977fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f9c981457f0 0x7f9c9813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:37136/0 (socket says 172.21.15.84:37136) 2024-03-29T10:42:29.688 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.703+0000 7f9c977fe640 1 -- 172.21.15.84:0/910037195 learned_addr learned my addr 172.21.15.84:0/910037195 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:42:29.688 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.703+0000 7f9c967fc640 1 -- 172.21.15.84:0/910037195 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3458791345 0 0) 0x7f9c98145eb0 con 0x7f9c9813f650 2024-03-29T10:42:29.688 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.703+0000 7f9c967fc640 10 monclient(hunting): _negotiate 2024-03-29T10:42:29.688 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.703+0000 7f9c967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:29.688 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.703+0000 7f9c967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:29.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.703+0000 7f9c967fc640 10 monclient(hunting): my global_id is 4559 2024-03-29T10:42:29.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.703+0000 7f9c967fc640 1 -- 172.21.15.84:0/910037195 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c80003520 con 0x7f9c9813f650 2024-03-29T10:42:29.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.703+0000 7f9c967fc640 1 -- 172.21.15.84:0/910037195 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2923502078 0 0) 0x7f9c98145bd0 con 0x7f9c981457f0 2024-03-29T10:42:29.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c967fc640 10 monclient(hunting): _negotiate 2024-03-29T10:42:29.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:29.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:29.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c967fc640 10 monclient(hunting): my global_id is 4447 2024-03-29T10:42:29.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c967fc640 1 -- 172.21.15.84:0/910037195 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c80003840 con 0x7f9c981457f0 2024-03-29T10:42:29.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c967fc640 1 -- 172.21.15.84:0/910037195 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2154040018 0 0) 0x7f9c80003520 con 0x7f9c9813f650 2024-03-29T10:42:29.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c967fc640 1 -- 172.21.15.84:0/910037195 >> v1:172.21.15.86:6789/0 conn(0x7f9c981457f0 legacy=0x7f9c9813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:29.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c967fc640 1 -- 172.21.15.84:0/910037195 >> v1:172.21.15.84:6789/0 conn(0x7f9c980a9d70 legacy=0x7f9c9814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:29.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c967fc640 10 monclient: _finish_hunting 0 2024-03-29T10:42:29.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c967fc640 1 monclient: found mon.noname-b 2024-03-29T10:42:29.689 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:29.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c967fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:42:29.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c967fc640 1 -- 172.21.15.84:0/910037195 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c98146340 con 0x7f9c9813f650 2024-03-29T10:42:29.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c967fc640 10 monclient: _finish_auth 0 2024-03-29T10:42:29.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c967fc640 10 monclient: _check_auth_tickets 2024-03-29T10:42:29.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:29.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c9de4a640 5 monclient: authenticate success, global_id 4559 2024-03-29T10:42:29.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c967fc640 1 -- 172.21.15.84:0/910037195 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f9c88001670 con 0x7f9c9813f650 2024-03-29T10:42:29.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:29.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:29.690 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c967fc640 10 monclient: dump: 2024-03-29T10:42:29.690 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:29.690 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:29.690 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:29.690 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:29.691 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:29.691 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:29.691 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:29.691 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:29.691 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:29.691 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:29.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c9de4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:29.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c967fc640 1 -- 172.21.15.84:0/910037195 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c880017d0 con 0x7f9c9813f650 2024-03-29T10:42:29.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:29.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c9de4a640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:29.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c967fc640 1 -- 172.21.15.84:0/910037195 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f9c88001be0 con 0x7f9c9813f650 2024-03-29T10:42:29.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:29.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c9de4a640 4 set_mon_vals no callback set 2024-03-29T10:42:29.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:29.691 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.704+0000 7f9c967fc640 10 monclient: dump: 2024-03-29T10:42:29.691 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:29.692 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:29.692 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:29.692 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:29.692 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:29.692 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:29.692 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:29.692 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:29.692 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:29.692 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:29.696 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.705+0000 7f9c9de4a640 10 monclient: shutdown 2024-03-29T10:42:29.696 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.705+0000 7f9c9de4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:29.696 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.705+0000 7f9c9de4a640 1 -- 172.21.15.84:0/910037195 >> v1:172.21.15.84:6790/0 conn(0x7f9c9813f650 legacy=0x7f9c980b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:29.696 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.705+0000 7f9c9de4a640 1 -- 172.21.15.84:0/910037195 shutdown_connections 2024-03-29T10:42:29.696 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.705+0000 7f9c9de4a640 1 -- 172.21.15.84:0/910037195 >> 172.21.15.84:0/910037195 conn(0x7f9c98066a00 msgr2=0x7f9c98066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:29.696 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.705+0000 7f9c9de4a640 1 -- 172.21.15.84:0/910037195 shutdown_connections 2024-03-29T10:42:29.696 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.705+0000 7f9c9de4a640 1 -- 172.21.15.84:0/910037195 wait complete. 2024-03-29T10:42:29.696 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.706+0000 7f9c9de4a640 10 monclient: build_initial_monmap 2024-03-29T10:42:29.696 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.706+0000 7f9c9de4a640 10 monclient: monmap: 2024-03-29T10:42:29.696 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:42:29.696 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:29.696 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:42:29.697 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:42:29.697 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:29.697 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:29.697 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:42:29.697 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:42:29.697 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:42:29.697 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:29.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.707+0000 7f9c9de4a640 1 librados: starting msgr at 2024-03-29T10:42:29.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.707+0000 7f9c9de4a640 1 librados: starting objecter 2024-03-29T10:42:29.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.707+0000 7f9c9de4a640 1 Processor -- start 2024-03-29T10:42:29.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.708+0000 7f9c9de4a640 1 -- start start 2024-03-29T10:42:29.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.708+0000 7f9c9de4a640 1 librados: setting wanted keys 2024-03-29T10:42:29.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.708+0000 7f9c9de4a640 1 librados: calling monclient init 2024-03-29T10:42:29.697 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.708+0000 7f9c9de4a640 10 monclient: init 2024-03-29T10:42:29.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.708+0000 7f9c9de4a640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:29.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.708+0000 7f9c9de4a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:42:29.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.708+0000 7f9c9de4a640 10 monclient(hunting): picked mon.noname-a con 0x7f9c980a9d70 addr v1:172.21.15.84:6789/0 2024-03-29T10:42:29.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.708+0000 7f9c9de4a640 10 monclient(hunting): picked mon.noname-b con 0x7f9c9813f650 addr v1:172.21.15.86:6789/0 2024-03-29T10:42:29.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.709+0000 7f9c9de4a640 10 monclient(hunting): picked mon.noname-c con 0x7f9c981457f0 addr v1:172.21.15.84:6790/0 2024-03-29T10:42:29.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.709+0000 7f9c9de4a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c9813b510 con 0x7f9c980a9d70 2024-03-29T10:42:29.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.709+0000 7f9c9de4a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c98138760 con 0x7f9c9813f650 2024-03-29T10:42:29.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.709+0000 7f9c9de4a640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c9813bd10 con 0x7f9c981457f0 2024-03-29T10:42:29.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.709+0000 7f9c9de4a640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:29.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.709+0000 7f9c9de4a640 10 monclient(hunting): authenticate will time out at 1251.972163s 2024-03-29T10:42:29.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.709+0000 7f9c97fff640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f9c981457f0 0x7f9c981cb340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:35944/0 (socket says 172.21.15.84:35944) 2024-03-29T10:42:29.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.709+0000 7f9c97fff640 1 -- 172.21.15.84:0/3491337554 learned_addr learned my addr 172.21.15.84:0/3491337554 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:42:29.698 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.709+0000 7f9c7ffff640 10 client.?.objecter ms_handle_connect 0x7f9c981457f0 2024-03-29T10:42:29.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.709+0000 7f9c7ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:29.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.709+0000 7f9c7ffff640 10 client.?.objecter ms_handle_connect 0x7f9c980a9d70 2024-03-29T10:42:29.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.709+0000 7f9c7ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:29.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.709+0000 7f9c7ffff640 10 client.?.objecter ms_handle_connect 0x7f9c9813f650 2024-03-29T10:42:29.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.709+0000 7f9c7ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:29.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.709+0000 7f9c7ffff640 1 -- 172.21.15.84:0/3491337554 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1020042973 0 0) 0x7f9c9813bd10 con 0x7f9c981457f0 2024-03-29T10:42:29.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.709+0000 7f9c7ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:42:29.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.709+0000 7f9c7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:29.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.709+0000 7f9c7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:29.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.709+0000 7f9c7ffff640 10 monclient(hunting): my global_id is 4562 2024-03-29T10:42:29.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.709+0000 7f9c7ffff640 1 -- 172.21.15.84:0/3491337554 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c84004eb0 con 0x7f9c981457f0 2024-03-29T10:42:29.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.710+0000 7f9c7ffff640 1 -- 172.21.15.84:0/3491337554 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1967351023 0 0) 0x7f9c98138760 con 0x7f9c9813f650 2024-03-29T10:42:29.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.710+0000 7f9c7ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:42:29.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.710+0000 7f9c7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:29.699 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.710+0000 7f9c7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:29.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.710+0000 7f9c7ffff640 10 monclient(hunting): my global_id is 4450 2024-03-29T10:42:29.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.710+0000 7f9c7ffff640 1 -- 172.21.15.84:0/3491337554 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c9813bd10 con 0x7f9c9813f650 2024-03-29T10:42:29.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.710+0000 7f9c7ffff640 1 -- 172.21.15.84:0/3491337554 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3445547756 0 0) 0x7f9c9813b510 con 0x7f9c980a9d70 2024-03-29T10:42:29.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.710+0000 7f9c7ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:42:29.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.710+0000 7f9c7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:29.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.710+0000 7f9c7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:29.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.710+0000 7f9c7ffff640 10 monclient(hunting): my global_id is 4479 2024-03-29T10:42:29.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.710+0000 7f9c7ffff640 1 -- 172.21.15.84:0/3491337554 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c98138760 con 0x7f9c980a9d70 2024-03-29T10:42:29.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.710+0000 7f9c7ffff640 1 -- 172.21.15.84:0/3491337554 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2042076461 0 0) 0x7f9c9813bd10 con 0x7f9c9813f650 2024-03-29T10:42:29.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.710+0000 7f9c7ffff640 1 -- 172.21.15.84:0/3491337554 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c9813b510 con 0x7f9c9813f650 2024-03-29T10:42:29.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.710+0000 7f9c7ffff640 1 -- 172.21.15.84:0/3491337554 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f9c88001ae0 con 0x7f9c9813f650 2024-03-29T10:42:29.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.710+0000 7f9c7ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:29.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.710+0000 7f9c7ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:29.700 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.710+0000 7f9c7ffff640 10 monclient(hunting): dump: 2024-03-29T10:42:29.701 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:29.701 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:29.701 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:29.701 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:29.701 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:29.701 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:29.701 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:29.701 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:29.701 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:29.701 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:29.701 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.710+0000 7f9c7ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:29.701 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.710+0000 7f9c7ffff640 1 -- 172.21.15.84:0/3491337554 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2110852884 0 0) 0x7f9c84004eb0 con 0x7f9c981457f0 2024-03-29T10:42:29.701 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.710+0000 7f9c7ffff640 1 -- 172.21.15.84:0/3491337554 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c9813bd10 con 0x7f9c981457f0 2024-03-29T10:42:29.701 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.710+0000 7f9c7ffff640 1 -- 172.21.15.84:0/3491337554 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f9c90001cd0 con 0x7f9c981457f0 2024-03-29T10:42:29.701 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.710+0000 7f9c7ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:29.702 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.710+0000 7f9c7ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:29.702 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.710+0000 7f9c7ffff640 10 monclient(hunting): dump: 2024-03-29T10:42:29.702 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:29.702 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:29.702 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:29.702 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:29.702 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:29.702 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:29.702 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:29.702 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:29.702 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:29.702 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:29.702 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.711+0000 7f9c7ffff640 1 -- 172.21.15.84:0/3491337554 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1390513685 0 0) 0x7f9c98138760 con 0x7f9c980a9d70 2024-03-29T10:42:29.702 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.711+0000 7f9c7ffff640 1 -- 172.21.15.84:0/3491337554 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c84004eb0 con 0x7f9c980a9d70 2024-03-29T10:42:29.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.711+0000 7f9c7ffff640 1 -- 172.21.15.84:0/3491337554 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f9c8c0017d0 con 0x7f9c980a9d70 2024-03-29T10:42:29.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.711+0000 7f9c7ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:29.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.711+0000 7f9c7ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:29.703 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.711+0000 7f9c7ffff640 10 monclient(hunting): dump: 2024-03-29T10:42:29.703 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:29.703 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:29.703 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:29.703 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:29.703 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:29.703 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:29.703 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:29.704 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:29.704 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:29.704 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:29.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.711+0000 7f9c7ffff640 1 -- 172.21.15.84:0/3491337554 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2519822546 0 0) 0x7f9c9813bd10 con 0x7f9c981457f0 2024-03-29T10:42:29.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.711+0000 7f9c7ffff640 1 -- 172.21.15.84:0/3491337554 >> v1:172.21.15.86:6789/0 conn(0x7f9c9813f650 legacy=0x7f9c981dbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:29.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.711+0000 7f9c7ffff640 1 -- 172.21.15.84:0/3491337554 >> v1:172.21.15.84:6789/0 conn(0x7f9c980a9d70 legacy=0x7f9c981ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:29.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.711+0000 7f9c7ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:42:29.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.711+0000 7f9c7ffff640 1 monclient: found mon.c 2024-03-29T10:42:29.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.711+0000 7f9c7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:29.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.711+0000 7f9c7ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:42:29.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.711+0000 7f9c7ffff640 1 -- 172.21.15.84:0/3491337554 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c98138de0 con 0x7f9c981457f0 2024-03-29T10:42:29.704 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.711+0000 7f9c7ffff640 10 monclient: _finish_auth 0 2024-03-29T10:42:29.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.711+0000 7f9c7ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:42:29.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.711+0000 7f9c7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:29.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.711+0000 7f9c9de4a640 5 monclient: authenticate success, global_id 4562 2024-03-29T10:42:29.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.711+0000 7f9c9de4a640 10 monclient: _renew_subs 2024-03-29T10:42:29.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.711+0000 7f9c9de4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:42:29.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.711+0000 7f9c9de4a640 1 -- 172.21.15.84:0/3491337554 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c981cef60 con 0x7f9c981457f0 2024-03-29T10:42:29.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.711+0000 7f9c9de4a640 10 client.4562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:29.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.711+0000 7f9c9de4a640 10 monclient: _renew_subs 2024-03-29T10:42:29.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.711+0000 7f9c9de4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:42:29.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.711+0000 7f9c9de4a640 1 -- 172.21.15.84:0/3491337554 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c981dc5d0 con 0x7f9c981457f0 2024-03-29T10:42:29.705 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.711+0000 7f9c9de4a640 1 librados: init done 2024-03-29T10:42:29.706 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.711+0000 7f9c7ffff640 1 -- 172.21.15.84:0/3491337554 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c90003990 con 0x7f9c981457f0 2024-03-29T10:42:29.706 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.712+0000 7f9c7ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:29.706 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.712+0000 7f9c7ffff640 1 -- 172.21.15.84:0/3491337554 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f9c90003da0 con 0x7f9c981457f0 2024-03-29T10:42:29.706 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.712+0000 7f9c7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:29.706 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.712+0000 7f9c7ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:29.706 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.712+0000 7f9c7ffff640 10 monclient: dump: 2024-03-29T10:42:29.706 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:29.706 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:29.706 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:29.706 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:29.707 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:29.707 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:29.707 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:29.707 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:29.707 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:29.707 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:29.707 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.712+0000 7f9c95ffb640 4 set_mon_vals no callback set 2024-03-29T10:42:29.707 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.713+0000 7f9c9de4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:29.707 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.713+0000 7f9c9de4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:29.707 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.713+0000 7f9c9de4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:42:29.707 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.713+0000 7f9c9de4a640 1 -- 172.21.15.84:0/3491337554 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c78005080 con 0x7f9c981457f0 2024-03-29T10:42:29.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.714+0000 7f9c7ffff640 1 -- 172.21.15.84:0/3491337554 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f9c90003f00 con 0x7f9c981457f0 2024-03-29T10:42:29.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.715+0000 7f9c977fe640 10 monclient: get_auth_request con 0x7f9c8406e140 auth_method 0 2024-03-29T10:42:29.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.715+0000 7f9c7ffff640 1 -- 172.21.15.84:0/3491337554 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f9c90084d70 con 0x7f9c981457f0 2024-03-29T10:42:29.708 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.715+0000 7f9c7ffff640 10 client.4562.objecter ms_dispatch 0x7f9c98000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:29.709 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.715+0000 7f9c7ffff640 3 client.4562.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:29.709 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.715+0000 7f9c7ffff640 3 client.4562.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:29.709 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.715+0000 7f9c7ffff640 20 client.4562.objecter dump_active .. 0 homeless 2024-03-29T10:42:29.709 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.715+0000 7f9c7ffff640 10 client.4562.objecter ms_handle_connect 0x7f9c8406e140 2024-03-29T10:42:29.709 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.721+0000 7f9c7ffff640 1 -- 172.21.15.84:0/3491337554 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9c90050650 con 0x7f9c981457f0 2024-03-29T10:42:29.709 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.721+0000 7f9c7ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:29.709 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:29.721+0000 7f9c7ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:30.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:30.086+0000 7f9c9de4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:30.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:30.086+0000 7f9c9de4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:30.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:30.086+0000 7f9c9de4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:42:30.070 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:30.086+0000 7f9c9de4a640 1 -- 172.21.15.84:0/3491337554 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c78005330 con 0x7f9c981457f0 2024-03-29T10:42:30.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:30.087+0000 7f9c7ffff640 1 -- 172.21.15.84:0/3491337554 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f9c90052580 con 0x7f9c981457f0 2024-03-29T10:42:30.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:30.087+0000 7f9c7ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:30.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:30.087+0000 7f9c7ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:30.071 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:42:30.071 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:30.074 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:30.091+0000 7f9c9de4a640 10 librados: watch_flush enter 2024-03-29T10:42:30.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:30.091+0000 7f9c9de4a640 10 librados: watch_flush exit 2024-03-29T10:42:30.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:30.091+0000 7f9c9de4a640 20 client.4562.objecter shutdown clearing up homeless session... 2024-03-29T10:42:30.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:30.091+0000 7f9c9de4a640 10 client.4562.objecter successfully canceled tick 2024-03-29T10:42:30.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:30.091+0000 7f9c9de4a640 1 -- 172.21.15.84:0/3491337554 >> v1:172.21.15.86:6816/3337225337 conn(0x7f9c8406e140 legacy=0x7f9c84070600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:30.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:30.091+0000 7f9c9de4a640 10 monclient: shutdown 2024-03-29T10:42:30.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:30.091+0000 7f9c9de4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:30.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:30.091+0000 7f9c9de4a640 1 -- 172.21.15.84:0/3491337554 >> v1:172.21.15.84:6790/0 conn(0x7f9c981457f0 legacy=0x7f9c981cb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:30.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:30.092+0000 7f9c9de4a640 1 -- 172.21.15.84:0/3491337554 shutdown_connections 2024-03-29T10:42:30.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:30.092+0000 7f9c9de4a640 1 -- 172.21.15.84:0/3491337554 >> 172.21.15.84:0/3491337554 conn(0x7f9c98066a00 msgr2=0x7f9c980b4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:30.079 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:30.092+0000 7f9c9de4a640 1 -- 172.21.15.84:0/3491337554 shutdown_connections 2024-03-29T10:42:30.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:30.092+0000 7f9c9de4a640 1 -- 172.21.15.84:0/3491337554 wait complete. 2024-03-29T10:42:30.080 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:30.092+0000 7f9c9de4a640 1 librados: shutdown 2024-03-29T10:42:30.090 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:36.897 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:37.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.061+0000 7fb4bf3b2640 10 monclient: get_monmap_and_config 2024-03-29T10:42:37.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.061+0000 7fb4bf3b2640 10 monclient: build_initial_monmap 2024-03-29T10:42:37.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.061+0000 7fb4bf3b2640 10 monclient: monmap: 2024-03-29T10:42:37.045 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:42:37.046 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:37.046 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:42:37.062618+0000 2024-03-29T10:42:37.046 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:42:37.062618+0000 2024-03-29T10:42:37.046 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:37.050 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:37.050 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:42:37.050 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:42:37.050 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:42:37.050 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:37.052 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.063+0000 7fb4bf3b2640 1 Processor -- start 2024-03-29T10:42:37.052 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.064+0000 7fb4bf3b2640 1 -- start start 2024-03-29T10:42:37.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.064+0000 7fb4bf3b2640 10 monclient: init 2024-03-29T10:42:37.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.065+0000 7fb4bf3b2640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:37.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.065+0000 7fb4bf3b2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:42:37.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.065+0000 7fb4bf3b2640 10 monclient(hunting): picked mon.noname-b con 0x7fb4b0006410 addr v1:172.21.15.84:6790/0 2024-03-29T10:42:37.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.065+0000 7fb4bf3b2640 10 monclient(hunting): picked mon.noname-a con 0x7fb4b002a640 addr v1:172.21.15.84:6789/0 2024-03-29T10:42:37.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.065+0000 7fb4bf3b2640 10 monclient(hunting): picked mon.noname-c con 0x7fb4b002b370 addr v1:172.21.15.86:6789/0 2024-03-29T10:42:37.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.065+0000 7fb4bf3b2640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b002a120 con 0x7fb4b002a640 2024-03-29T10:42:37.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.065+0000 7fb4bf3b2640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b00c75b0 con 0x7fb4b002b370 2024-03-29T10:42:37.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.065+0000 7fb4bf3b2640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b0033110 con 0x7fb4b0006410 2024-03-29T10:42:37.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.065+0000 7fb4bf3b2640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:37.053 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.065+0000 7fb4bf3b2640 10 monclient(hunting): authenticate will time out at 1259.328524s 2024-03-29T10:42:37.056 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.067+0000 7fb4be3b0640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fb4b0006410 0x7fb4b00345e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:33924/0 (socket says 172.21.15.84:33924) 2024-03-29T10:42:37.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.067+0000 7fb4be3b0640 1 -- 172.21.15.84:0/874677122 learned_addr learned my addr 172.21.15.84:0/874677122 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:42:37.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.067+0000 7fb4bd3ae640 1 -- 172.21.15.84:0/874677122 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051159992 0 0) 0x7fb4b0033110 con 0x7fb4b0006410 2024-03-29T10:42:37.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.067+0000 7fb4bd3ae640 10 monclient(hunting): _negotiate 2024-03-29T10:42:37.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.067+0000 7fb4bd3ae640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:37.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.067+0000 7fb4bd3ae640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:37.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.067+0000 7fb4bd3ae640 10 monclient(hunting): my global_id is 4565 2024-03-29T10:42:37.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.067+0000 7fb4bd3ae640 1 -- 172.21.15.84:0/874677122 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4a4003520 con 0x7fb4b0006410 2024-03-29T10:42:37.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.068+0000 7fb4bd3ae640 1 -- 172.21.15.84:0/874677122 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 541659413 0 0) 0x7fb4a4003520 con 0x7fb4b0006410 2024-03-29T10:42:37.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.068+0000 7fb4bd3ae640 1 -- 172.21.15.84:0/874677122 >> v1:172.21.15.86:6789/0 conn(0x7fb4b002b370 legacy=0x7fb4b00320c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:37.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.068+0000 7fb4bd3ae640 1 -- 172.21.15.84:0/874677122 >> v1:172.21.15.84:6789/0 conn(0x7fb4b002a640 legacy=0x7fb4b00c6ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:37.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.070+0000 7fb4bd3ae640 10 monclient: _finish_hunting 0 2024-03-29T10:42:37.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.070+0000 7fb4bd3ae640 1 monclient: found mon.noname-b 2024-03-29T10:42:37.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.070+0000 7fb4bd3ae640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:37.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.070+0000 7fb4bd3ae640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:42:37.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.070+0000 7fb4bd3ae640 1 -- 172.21.15.84:0/874677122 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b00332a0 con 0x7fb4b0006410 2024-03-29T10:42:37.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.070+0000 7fb4bd3ae640 10 monclient: _finish_auth 0 2024-03-29T10:42:37.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.070+0000 7fb4bd3ae640 10 monclient: _check_auth_tickets 2024-03-29T10:42:37.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.070+0000 7fb4bd3ae640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:37.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.070+0000 7fb4bd3ae640 1 -- 172.21.15.84:0/874677122 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fb4a8001690 con 0x7fb4b0006410 2024-03-29T10:42:37.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.070+0000 7fb4bd3ae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:37.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.070+0000 7fb4bd3ae640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:37.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.070+0000 7fb4bd3ae640 10 monclient: dump: 2024-03-29T10:42:37.058 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:37.058 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:37.058 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:37.059 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:37.059 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:37.059 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:37.059 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:37.059 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:37.059 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:37.059 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:37.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.070+0000 7fb4bf3b2640 5 monclient: authenticate success, global_id 4565 2024-03-29T10:42:37.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.070+0000 7fb4bf3b2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:37.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.071+0000 7fb4bd3ae640 1 -- 172.21.15.84:0/874677122 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4a80017f0 con 0x7fb4b0006410 2024-03-29T10:42:37.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.071+0000 7fb4bd3ae640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:37.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.071+0000 7fb4bd3ae640 1 -- 172.21.15.84:0/874677122 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fb4a8001c40 con 0x7fb4b0006410 2024-03-29T10:42:37.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.071+0000 7fb4bd3ae640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:37.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.071+0000 7fb4bd3ae640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:37.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.071+0000 7fb4bd3ae640 10 monclient: dump: 2024-03-29T10:42:37.060 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:37.060 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:37.060 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:37.060 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:37.060 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:37.060 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:37.060 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:37.060 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:37.060 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:37.060 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:37.060 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.071+0000 7fb4bf3b2640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:37.060 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.071+0000 7fb4bf3b2640 4 set_mon_vals no callback set 2024-03-29T10:42:37.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.072+0000 7fb4bf3b2640 10 monclient: shutdown 2024-03-29T10:42:37.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.072+0000 7fb4bf3b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:37.083 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.072+0000 7fb4bf3b2640 1 -- 172.21.15.84:0/874677122 >> v1:172.21.15.84:6790/0 conn(0x7fb4b0006410 legacy=0x7fb4b00345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:37.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.073+0000 7fb4bf3b2640 1 -- 172.21.15.84:0/874677122 shutdown_connections 2024-03-29T10:42:37.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.073+0000 7fb4bf3b2640 1 -- 172.21.15.84:0/874677122 >> 172.21.15.84:0/874677122 conn(0x7fb4b00058f0 msgr2=0x7fb4b0005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:37.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.074+0000 7fb4bf3b2640 1 -- 172.21.15.84:0/874677122 shutdown_connections 2024-03-29T10:42:37.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.076+0000 7fb4bf3b2640 1 -- 172.21.15.84:0/874677122 wait complete. 2024-03-29T10:42:37.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.077+0000 7fb4bf3b2640 10 monclient: build_initial_monmap 2024-03-29T10:42:37.084 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.077+0000 7fb4bf3b2640 10 monclient: monmap: 2024-03-29T10:42:37.084 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:42:37.084 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:37.084 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:42:37.084 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:42:37.084 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:37.084 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:37.084 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:42:37.085 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:42:37.085 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:42:37.085 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:37.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.077+0000 7fb4bf3b2640 1 librados: starting msgr at 2024-03-29T10:42:37.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.077+0000 7fb4bf3b2640 1 librados: starting objecter 2024-03-29T10:42:37.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.080+0000 7fb4bf3b2640 1 Processor -- start 2024-03-29T10:42:37.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.080+0000 7fb4bf3b2640 1 -- start start 2024-03-29T10:42:37.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.080+0000 7fb4bf3b2640 1 librados: setting wanted keys 2024-03-29T10:42:37.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.080+0000 7fb4bf3b2640 1 librados: calling monclient init 2024-03-29T10:42:37.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.080+0000 7fb4bf3b2640 10 monclient: init 2024-03-29T10:42:37.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.081+0000 7fb4bf3b2640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:37.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.081+0000 7fb4bf3b2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:42:37.085 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.081+0000 7fb4bf3b2640 10 monclient(hunting): picked mon.noname-a con 0x7fb4b0006410 addr v1:172.21.15.84:6789/0 2024-03-29T10:42:37.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.081+0000 7fb4bf3b2640 10 monclient(hunting): picked mon.noname-b con 0x7fb4b002a640 addr v1:172.21.15.86:6789/0 2024-03-29T10:42:37.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.082+0000 7fb4bf3b2640 10 monclient(hunting): picked mon.noname-c con 0x7fb4b002b370 addr v1:172.21.15.84:6790/0 2024-03-29T10:42:37.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.082+0000 7fb4bf3b2640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b00c46a0 con 0x7fb4b0006410 2024-03-29T10:42:37.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.082+0000 7fb4bf3b2640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b00c4980 con 0x7fb4b002a640 2024-03-29T10:42:37.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.082+0000 7fb4bf3b2640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b00bd800 con 0x7fb4b002b370 2024-03-29T10:42:37.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.082+0000 7fb4bf3b2640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:37.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.082+0000 7fb4bf3b2640 10 monclient(hunting): authenticate will time out at 1259.345443s 2024-03-29T10:42:37.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.082+0000 7fb4bebb1640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fb4b002b370 0x7fb4b00c5520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:35662/0 (socket says 172.21.15.84:35662) 2024-03-29T10:42:37.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.082+0000 7fb4bebb1640 1 -- 172.21.15.84:0/63095571 learned_addr learned my addr 172.21.15.84:0/63095571 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:42:37.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.082+0000 7fb4aeffd640 10 client.?.objecter ms_handle_connect 0x7fb4b002b370 2024-03-29T10:42:37.086 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.082+0000 7fb4aeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:37.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.083+0000 7fb4aeffd640 1 -- 172.21.15.84:0/63095571 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291935946 0 0) 0x7fb4b00bd800 con 0x7fb4b002b370 2024-03-29T10:42:37.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.083+0000 7fb4aeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:37.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.083+0000 7fb4aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:37.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.083+0000 7fb4aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:37.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.083+0000 7fb4aeffd640 10 monclient(hunting): my global_id is 4568 2024-03-29T10:42:37.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.083+0000 7fb4aeffd640 1 -- 172.21.15.84:0/63095571 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb49c004eb0 con 0x7fb4b002b370 2024-03-29T10:42:37.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.086+0000 7fb4aeffd640 1 -- 172.21.15.84:0/63095571 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 25522450 0 0) 0x7fb49c004eb0 con 0x7fb4b002b370 2024-03-29T10:42:37.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.086+0000 7fb4aeffd640 1 -- 172.21.15.84:0/63095571 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4b00bd800 con 0x7fb4b002b370 2024-03-29T10:42:37.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.086+0000 7fb4aeffd640 1 -- 172.21.15.84:0/63095571 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fb4b804fdd0 con 0x7fb4b002b370 2024-03-29T10:42:37.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.086+0000 7fb4aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:37.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.086+0000 7fb4aeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:37.087 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.086+0000 7fb4aeffd640 10 monclient(hunting): dump: 2024-03-29T10:42:37.087 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:37.088 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:37.088 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:37.088 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:37.088 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:37.088 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:37.088 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:37.088 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:37.088 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:37.088 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:37.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.086+0000 7fb4aeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:37.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.086+0000 7fb4aeffd640 10 client.?.objecter ms_handle_connect 0x7fb4b002a640 2024-03-29T10:42:37.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.086+0000 7fb4aeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:37.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.086+0000 7fb4aeffd640 1 -- 172.21.15.84:0/63095571 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2701895474 0 0) 0x7fb4b00c4980 con 0x7fb4b002a640 2024-03-29T10:42:37.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.087+0000 7fb4aeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:37.088 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.087+0000 7fb4aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:37.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.087+0000 7fb4aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:37.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.087+0000 7fb4aeffd640 10 monclient(hunting): my global_id is 4453 2024-03-29T10:42:37.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.087+0000 7fb4aeffd640 1 -- 172.21.15.84:0/63095571 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb49c004eb0 con 0x7fb4b002a640 2024-03-29T10:42:37.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.087+0000 7fb4aeffd640 1 -- 172.21.15.84:0/63095571 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1119623857 0 0) 0x7fb49c004eb0 con 0x7fb4b002a640 2024-03-29T10:42:37.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.087+0000 7fb4aeffd640 1 -- 172.21.15.84:0/63095571 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4b00c4980 con 0x7fb4b002a640 2024-03-29T10:42:37.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.087+0000 7fb4aeffd640 1 -- 172.21.15.84:0/63095571 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fb4b4001830 con 0x7fb4b002a640 2024-03-29T10:42:37.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.087+0000 7fb4aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:37.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.087+0000 7fb4aeffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:37.089 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.087+0000 7fb4aeffd640 10 monclient(hunting): dump: 2024-03-29T10:42:37.089 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:37.089 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:37.089 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:37.090 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:37.090 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:37.090 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:37.090 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:37.090 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:37.090 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:37.090 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:37.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.089+0000 7fb4aeffd640 1 -- 172.21.15.84:0/63095571 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1219427264 0 0) 0x7fb4b00c4980 con 0x7fb4b002a640 2024-03-29T10:42:37.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.089+0000 7fb4aeffd640 1 -- 172.21.15.84:0/63095571 >> v1:172.21.15.84:6790/0 conn(0x7fb4b002b370 legacy=0x7fb4b00c5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:37.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.089+0000 7fb4aeffd640 1 -- 172.21.15.84:0/63095571 >> v1:172.21.15.84:6789/0 conn(0x7fb4b0006410 legacy=0x7fb4b00c2e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:42:37.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.090+0000 7fb4aeffd640 10 monclient: _finish_hunting 0 2024-03-29T10:42:37.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.090+0000 7fb4aeffd640 1 monclient: found mon.b 2024-03-29T10:42:37.090 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.090+0000 7fb4aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:37.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.090+0000 7fb4aeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:42:37.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.090+0000 7fb4aeffd640 1 -- 172.21.15.84:0/63095571 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b00c5c30 con 0x7fb4b002a640 2024-03-29T10:42:37.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.090+0000 7fb4aeffd640 10 monclient: _finish_auth 0 2024-03-29T10:42:37.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.090+0000 7fb4aeffd640 10 monclient: _check_auth_tickets 2024-03-29T10:42:37.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.090+0000 7fb4aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:37.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.090+0000 7fb4bf3b2640 5 monclient: authenticate success, global_id 4453 2024-03-29T10:42:37.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.090+0000 7fb4bf3b2640 10 monclient: _renew_subs 2024-03-29T10:42:37.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.090+0000 7fb4bf3b2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:42:37.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.090+0000 7fb4bf3b2640 1 -- 172.21.15.84:0/63095571 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4b0030020 con 0x7fb4b002a640 2024-03-29T10:42:37.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.090+0000 7fb4bf3b2640 10 client.4453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:37.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.091+0000 7fb4bf3b2640 10 monclient: _renew_subs 2024-03-29T10:42:37.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.091+0000 7fb4bf3b2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:42:37.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.091+0000 7fb4bf3b2640 1 -- 172.21.15.84:0/63095571 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b00c36f0 con 0x7fb4b002a640 2024-03-29T10:42:37.091 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.091+0000 7fb4bf3b2640 1 librados: init done 2024-03-29T10:42:37.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.091+0000 7fb4aeffd640 1 -- 172.21.15.84:0/63095571 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4b4001d40 con 0x7fb4b002a640 2024-03-29T10:42:37.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.091+0000 7fb4aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:37.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.091+0000 7fb4aeffd640 1 -- 172.21.15.84:0/63095571 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fb4b4003790 con 0x7fb4b002a640 2024-03-29T10:42:37.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.091+0000 7fb4bcbad640 4 set_mon_vals no callback set 2024-03-29T10:42:37.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.091+0000 7fb4aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:37.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.091+0000 7fb4aeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:37.092 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.091+0000 7fb4aeffd640 10 monclient: dump: 2024-03-29T10:42:37.092 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:37.092 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:37.092 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:37.092 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:37.092 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:37.092 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:37.092 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:37.092 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:37.093 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:37.093 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:37.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.095+0000 7fb4bf3b2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:37.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.095+0000 7fb4bf3b2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:37.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.095+0000 7fb4bf3b2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:42:37.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.095+0000 7fb4bf3b2640 1 -- 172.21.15.84:0/63095571 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a0005080 con 0x7fb4b002a640 2024-03-29T10:42:37.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.095+0000 7fb4aeffd640 1 -- 172.21.15.84:0/63095571 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7fb4b4019760 con 0x7fb4b002a640 2024-03-29T10:42:37.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.096+0000 7fb4aeffd640 1 -- 172.21.15.84:0/63095571 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7fb4b4083740 con 0x7fb4b002a640 2024-03-29T10:42:37.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.096+0000 7fb4aeffd640 10 client.4453.objecter ms_dispatch 0x7fb4b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:37.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.096+0000 7fb4aeffd640 3 client.4453.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:37.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.096+0000 7fb4aeffd640 3 client.4453.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:37.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.096+0000 7fb4aeffd640 20 client.4453.objecter dump_active .. 0 homeless 2024-03-29T10:42:37.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.097+0000 7fb4be3b0640 10 monclient: get_auth_request con 0x7fb49c06dce0 auth_method 0 2024-03-29T10:42:37.093 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.097+0000 7fb4aeffd640 10 client.4453.objecter ms_handle_connect 0x7fb49c06dce0 2024-03-29T10:42:37.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.104+0000 7fb4aeffd640 1 -- 172.21.15.84:0/63095571 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb4b404ee50 con 0x7fb4b002a640 2024-03-29T10:42:37.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.104+0000 7fb4aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:37.094 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.104+0000 7fb4aeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:37.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.465+0000 7fb4bf3b2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:37.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.465+0000 7fb4bf3b2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:37.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.465+0000 7fb4bf3b2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:42:37.449 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.465+0000 7fb4bf3b2640 1 -- 172.21.15.84:0/63095571 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4a0005330 con 0x7fb4b002a640 2024-03-29T10:42:37.450 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.467+0000 7fb4aeffd640 1 -- 172.21.15.84:0/63095571 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7fb4b4050f50 con 0x7fb4b002a640 2024-03-29T10:42:37.451 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:42:37.451 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:37.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.467+0000 7fb4aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:37.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.467+0000 7fb4aeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:37.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.470+0000 7fb4bf3b2640 10 librados: watch_flush enter 2024-03-29T10:42:37.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.470+0000 7fb4bf3b2640 10 librados: watch_flush exit 2024-03-29T10:42:37.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.470+0000 7fb4bf3b2640 20 client.4453.objecter shutdown clearing up homeless session... 2024-03-29T10:42:37.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.470+0000 7fb4bf3b2640 10 client.4453.objecter successfully canceled tick 2024-03-29T10:42:37.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.470+0000 7fb4bf3b2640 1 -- 172.21.15.84:0/63095571 >> v1:172.21.15.86:6816/3337225337 conn(0x7fb49c06dce0 legacy=0x7fb49c0701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:37.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.470+0000 7fb4bf3b2640 10 monclient: shutdown 2024-03-29T10:42:37.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.470+0000 7fb4bf3b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:37.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.470+0000 7fb4bf3b2640 1 -- 172.21.15.84:0/63095571 >> v1:172.21.15.86:6789/0 conn(0x7fb4b002a640 legacy=0x7fb4b00c4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:37.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.471+0000 7fb4bf3b2640 1 -- 172.21.15.84:0/63095571 shutdown_connections 2024-03-29T10:42:37.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.471+0000 7fb4bf3b2640 1 -- 172.21.15.84:0/63095571 >> 172.21.15.84:0/63095571 conn(0x7fb4b00058f0 msgr2=0x7fb4b00043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:37.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.471+0000 7fb4bf3b2640 1 -- 172.21.15.84:0/63095571 shutdown_connections 2024-03-29T10:42:37.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.471+0000 7fb4bf3b2640 1 -- 172.21.15.84:0/63095571 wait complete. 2024-03-29T10:42:37.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:37.471+0000 7fb4bf3b2640 1 librados: shutdown 2024-03-29T10:42:37.469 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:44.276 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:44.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.442+0000 7fec50833640 10 monclient: get_monmap_and_config 2024-03-29T10:42:44.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.442+0000 7fec50833640 10 monclient: build_initial_monmap 2024-03-29T10:42:44.425 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.442+0000 7fec50833640 10 monclient: monmap: 2024-03-29T10:42:44.425 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:42:44.425 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:44.426 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:42:44.443032+0000 2024-03-29T10:42:44.426 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:42:44.443032+0000 2024-03-29T10:42:44.426 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:44.426 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:44.426 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:42:44.426 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:42:44.426 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:42:44.426 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:44.426 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.443+0000 7fec50833640 1 Processor -- start 2024-03-29T10:42:44.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.443+0000 7fec50833640 1 -- start start 2024-03-29T10:42:44.427 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.443+0000 7fec50833640 10 monclient: init 2024-03-29T10:42:44.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.444+0000 7fec50833640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:44.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.444+0000 7fec50833640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:42:44.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.444+0000 7fec50833640 10 monclient(hunting): picked mon.noname-b con 0x7fec4c0b0860 addr v1:172.21.15.84:6790/0 2024-03-29T10:42:44.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.444+0000 7fec50833640 10 monclient(hunting): picked mon.noname-a con 0x7fec4c13ef30 addr v1:172.21.15.84:6789/0 2024-03-29T10:42:44.429 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.444+0000 7fec50833640 10 monclient(hunting): picked mon.noname-c con 0x7fec4c0af8e0 addr v1:172.21.15.86:6789/0 2024-03-29T10:42:44.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.444+0000 7fec50833640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec4c0b0c40 con 0x7fec4c13ef30 2024-03-29T10:42:44.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.444+0000 7fec50833640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec4c0aed10 con 0x7fec4c0af8e0 2024-03-29T10:42:44.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.444+0000 7fec50833640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec4c0afd50 con 0x7fec4c0b0860 2024-03-29T10:42:44.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.444+0000 7fec50833640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:44.430 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.444+0000 7fec50833640 10 monclient(hunting): authenticate will time out at 1266.708090s 2024-03-29T10:42:44.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.446+0000 7fec4a59c640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fec4c0b0860 0x7fec4c13e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:47538/0 (socket says 172.21.15.84:47538) 2024-03-29T10:42:44.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.446+0000 7fec4a59c640 1 -- 172.21.15.84:0/646711862 learned_addr learned my addr 172.21.15.84:0/646711862 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:42:44.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.447+0000 7fec4959a640 1 -- 172.21.15.84:0/646711862 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2979999404 0 0) 0x7fec4c0afd50 con 0x7fec4c0b0860 2024-03-29T10:42:44.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.447+0000 7fec4959a640 10 monclient(hunting): _negotiate 2024-03-29T10:42:44.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.447+0000 7fec4959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:44.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.447+0000 7fec4959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:44.433 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.447+0000 7fec4959a640 10 monclient(hunting): my global_id is 4571 2024-03-29T10:42:44.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.447+0000 7fec4959a640 1 -- 172.21.15.84:0/646711862 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec34003500 con 0x7fec4c0b0860 2024-03-29T10:42:44.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.447+0000 7fec4959a640 1 -- 172.21.15.84:0/646711862 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1981655407 0 0) 0x7fec34003500 con 0x7fec4c0b0860 2024-03-29T10:42:44.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.447+0000 7fec4959a640 1 -- 172.21.15.84:0/646711862 >> v1:172.21.15.86:6789/0 conn(0x7fec4c0af8e0 legacy=0x7fec4c149a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:44.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.449+0000 7fec4959a640 1 -- 172.21.15.84:0/646711862 >> v1:172.21.15.84:6789/0 conn(0x7fec4c13ef30 legacy=0x7fec4c0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:44.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.449+0000 7fec4959a640 10 monclient: _finish_hunting 0 2024-03-29T10:42:44.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.449+0000 7fec4959a640 1 monclient: found mon.noname-b 2024-03-29T10:42:44.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.449+0000 7fec4959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:44.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.449+0000 7fec4959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:42:44.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.449+0000 7fec4959a640 1 -- 172.21.15.84:0/646711862 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec4c0ab1b0 con 0x7fec4c0b0860 2024-03-29T10:42:44.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.449+0000 7fec4959a640 10 monclient: _finish_auth 0 2024-03-29T10:42:44.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.449+0000 7fec4959a640 10 monclient: _check_auth_tickets 2024-03-29T10:42:44.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.449+0000 7fec4959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:44.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.449+0000 7fec4959a640 1 -- 172.21.15.84:0/646711862 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fec3c004cb0 con 0x7fec4c0b0860 2024-03-29T10:42:44.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.449+0000 7fec4959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:44.434 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.449+0000 7fec4959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:44.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.449+0000 7fec4959a640 10 monclient: dump: 2024-03-29T10:42:44.435 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:44.435 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:44.435 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:44.435 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:44.435 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:44.435 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:44.435 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:44.435 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:44.435 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:44.435 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:44.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.449+0000 7fec4959a640 1 -- 172.21.15.84:0/646711862 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec3c004f40 con 0x7fec4c0b0860 2024-03-29T10:42:44.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.449+0000 7fec4959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:44.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.449+0000 7fec4959a640 1 -- 172.21.15.84:0/646711862 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fec3c001470 con 0x7fec4c0b0860 2024-03-29T10:42:44.435 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.449+0000 7fec4959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:44.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.449+0000 7fec4959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:44.436 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.449+0000 7fec4959a640 10 monclient: dump: 2024-03-29T10:42:44.436 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:44.436 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:44.436 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:44.436 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:44.436 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:44.436 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:44.436 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:44.436 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:44.436 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:44.436 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:44.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.451+0000 7fec50833640 5 monclient: authenticate success, global_id 4571 2024-03-29T10:42:44.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.451+0000 7fec50833640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:44.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.451+0000 7fec50833640 4 set_mon_vals no callback set 2024-03-29T10:42:44.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.452+0000 7fec50833640 10 monclient: shutdown 2024-03-29T10:42:44.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.452+0000 7fec50833640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:44.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.452+0000 7fec50833640 1 -- 172.21.15.84:0/646711862 >> v1:172.21.15.84:6790/0 conn(0x7fec4c0b0860 legacy=0x7fec4c13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:44.437 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.454+0000 7fec50833640 1 -- 172.21.15.84:0/646711862 shutdown_connections 2024-03-29T10:42:44.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.454+0000 7fec50833640 1 -- 172.21.15.84:0/646711862 >> 172.21.15.84:0/646711862 conn(0x7fec4c06b660 msgr2=0x7fec4c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:44.438 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.455+0000 7fec50833640 1 -- 172.21.15.84:0/646711862 shutdown_connections 2024-03-29T10:42:44.441 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.458+0000 7fec50833640 1 -- 172.21.15.84:0/646711862 wait complete. 2024-03-29T10:42:44.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.459+0000 7fec50833640 10 monclient: build_initial_monmap 2024-03-29T10:42:44.442 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.459+0000 7fec50833640 10 monclient: monmap: 2024-03-29T10:42:44.442 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:42:44.442 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:44.442 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:42:44.442 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:42:44.443 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:44.443 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:44.443 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:42:44.443 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:42:44.443 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:42:44.443 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:44.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.459+0000 7fec50833640 1 librados: starting msgr at 2024-03-29T10:42:44.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.459+0000 7fec50833640 1 librados: starting objecter 2024-03-29T10:42:44.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.460+0000 7fec50833640 1 Processor -- start 2024-03-29T10:42:44.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.460+0000 7fec50833640 1 -- start start 2024-03-29T10:42:44.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.460+0000 7fec50833640 1 librados: setting wanted keys 2024-03-29T10:42:44.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.460+0000 7fec50833640 1 librados: calling monclient init 2024-03-29T10:42:44.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.460+0000 7fec50833640 10 monclient: init 2024-03-29T10:42:44.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.460+0000 7fec50833640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:44.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.460+0000 7fec50833640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:42:44.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.461+0000 7fec50833640 10 monclient(hunting): picked mon.noname-b con 0x7fec4c0af8e0 addr v1:172.21.15.86:6789/0 2024-03-29T10:42:44.445 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.461+0000 7fec50833640 10 monclient(hunting): picked mon.noname-a con 0x7fec4c0b0860 addr v1:172.21.15.84:6789/0 2024-03-29T10:42:44.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.461+0000 7fec50833640 10 monclient(hunting): picked mon.noname-c con 0x7fec4c13ef30 addr v1:172.21.15.84:6790/0 2024-03-29T10:42:44.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.461+0000 7fec50833640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec4c0b4440 con 0x7fec4c0b0860 2024-03-29T10:42:44.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.461+0000 7fec50833640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec4c1db820 con 0x7fec4c0af8e0 2024-03-29T10:42:44.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.461+0000 7fec50833640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec4c0b3ce0 con 0x7fec4c13ef30 2024-03-29T10:42:44.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.461+0000 7fec50833640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:44.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.461+0000 7fec50833640 10 monclient(hunting): authenticate will time out at 1266.724180s 2024-03-29T10:42:44.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.463+0000 7fec4a59c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fec4c0af8e0 0x7fec4c1df070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:60571/0 (socket says 172.21.15.84:60571) 2024-03-29T10:42:44.446 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.463+0000 7fec4a59c640 1 -- 172.21.15.84:0/114701333 learned_addr learned my addr 172.21.15.84:0/114701333 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:42:44.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.464+0000 7fec3affd640 10 client.?.objecter ms_handle_connect 0x7fec4c0b0860 2024-03-29T10:42:44.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.464+0000 7fec3affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:44.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.465+0000 7fec3affd640 1 -- 172.21.15.84:0/114701333 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 163972664 0 0) 0x7fec4c0b4440 con 0x7fec4c0b0860 2024-03-29T10:42:44.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.465+0000 7fec3affd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:44.452 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.465+0000 7fec3affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:44.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.465+0000 7fec3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:44.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.465+0000 7fec3affd640 10 monclient(hunting): my global_id is 4485 2024-03-29T10:42:44.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.465+0000 7fec3affd640 1 -- 172.21.15.84:0/114701333 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec2c004eb0 con 0x7fec4c0b0860 2024-03-29T10:42:44.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.465+0000 7fec3affd640 10 client.?.objecter ms_handle_connect 0x7fec4c0af8e0 2024-03-29T10:42:44.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.465+0000 7fec3affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:44.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.465+0000 7fec3affd640 1 -- 172.21.15.84:0/114701333 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 410447476 0 0) 0x7fec4c1db820 con 0x7fec4c0af8e0 2024-03-29T10:42:44.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.465+0000 7fec3affd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:44.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.465+0000 7fec3affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:44.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.465+0000 7fec3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:44.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.465+0000 7fec3affd640 10 monclient(hunting): my global_id is 4456 2024-03-29T10:42:44.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.465+0000 7fec3affd640 1 -- 172.21.15.84:0/114701333 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec2c0051d0 con 0x7fec4c0af8e0 2024-03-29T10:42:44.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.466+0000 7fec3affd640 1 -- 172.21.15.84:0/114701333 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 79958192 0 0) 0x7fec2c004eb0 con 0x7fec4c0b0860 2024-03-29T10:42:44.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.466+0000 7fec3affd640 1 -- 172.21.15.84:0/114701333 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec4c1db820 con 0x7fec4c0b0860 2024-03-29T10:42:44.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.466+0000 7fec3affd640 1 -- 172.21.15.84:0/114701333 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fec400017c0 con 0x7fec4c0b0860 2024-03-29T10:42:44.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.466+0000 7fec3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:44.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.466+0000 7fec3affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:42:44.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.466+0000 7fec3affd640 10 monclient(hunting): dump: 2024-03-29T10:42:44.455 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:44.455 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:44.455 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:44.455 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:44.455 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:44.455 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:44.455 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:44.455 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:44.455 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:44.455 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:44.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.466+0000 7fec3affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:44.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.466+0000 7fec3affd640 1 -- 172.21.15.84:0/114701333 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2847591948 0 0) 0x7fec4c1db820 con 0x7fec4c0b0860 2024-03-29T10:42:44.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.466+0000 7fec3affd640 1 -- 172.21.15.84:0/114701333 >> v1:172.21.15.84:6790/0 conn(0x7fec4c13ef30 legacy=0x7fec4c1db110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:44.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.466+0000 7fec3affd640 1 -- 172.21.15.84:0/114701333 >> v1:172.21.15.86:6789/0 conn(0x7fec4c0af8e0 legacy=0x7fec4c1df070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:44.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.466+0000 7fec3affd640 10 monclient: _finish_hunting 0 2024-03-29T10:42:44.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.466+0000 7fec3affd640 1 monclient: found mon.a 2024-03-29T10:42:44.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.466+0000 7fec3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:44.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.466+0000 7fec3affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:42:44.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.466+0000 7fec3affd640 1 -- 172.21.15.84:0/114701333 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec4c0b2b90 con 0x7fec4c0b0860 2024-03-29T10:42:44.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.466+0000 7fec3affd640 10 monclient: _finish_auth 0 2024-03-29T10:42:44.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.466+0000 7fec3affd640 10 monclient: _check_auth_tickets 2024-03-29T10:42:44.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.466+0000 7fec3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:44.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.466+0000 7fec3affd640 1 -- 172.21.15.84:0/114701333 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec40001cb0 con 0x7fec4c0b0860 2024-03-29T10:42:44.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.466+0000 7fec3affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:44.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.466+0000 7fec3bfff640 4 set_mon_vals no callback set 2024-03-29T10:42:44.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.466+0000 7fec50833640 5 monclient: authenticate success, global_id 4485 2024-03-29T10:42:44.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.466+0000 7fec50833640 10 monclient: _renew_subs 2024-03-29T10:42:44.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.466+0000 7fec50833640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:42:44.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.466+0000 7fec50833640 1 -- 172.21.15.84:0/114701333 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec4c0b2e40 con 0x7fec4c0b0860 2024-03-29T10:42:44.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.467+0000 7fec50833640 10 client.4485.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:44.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.467+0000 7fec50833640 10 monclient: _renew_subs 2024-03-29T10:42:44.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.467+0000 7fec50833640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:42:44.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.467+0000 7fec50833640 1 -- 172.21.15.84:0/114701333 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec4c137bf0 con 0x7fec4c0b0860 2024-03-29T10:42:44.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.467+0000 7fec50833640 1 librados: init done 2024-03-29T10:42:44.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.467+0000 7fec3affd640 1 -- 172.21.15.84:0/114701333 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fec400017c0 con 0x7fec4c0b0860 2024-03-29T10:42:44.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.467+0000 7fec3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:44.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.467+0000 7fec3affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:42:44.457 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.467+0000 7fec3affd640 10 monclient: dump: 2024-03-29T10:42:44.458 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:44.458 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:44.458 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:44.458 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:44.458 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:44.458 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:44.458 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:44.458 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:44.458 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:44.458 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:44.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.468+0000 7fec50833640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:44.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.468+0000 7fec50833640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:44.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.468+0000 7fec50833640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:42:44.458 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.468+0000 7fec50833640 1 -- 172.21.15.84:0/114701333 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec4c0b3f20 con 0x7fec4c0b0860 2024-03-29T10:42:44.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.472+0000 7fec3affd640 1 -- 172.21.15.84:0/114701333 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7fec400194e0 con 0x7fec4c0b0860 2024-03-29T10:42:44.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.473+0000 7fec3affd640 1 -- 172.21.15.84:0/114701333 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7fec40083010 con 0x7fec4c0b0860 2024-03-29T10:42:44.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.473+0000 7fec3affd640 10 client.4485.objecter ms_dispatch 0x7fec4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:44.459 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.473+0000 7fec3affd640 3 client.4485.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:44.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.473+0000 7fec3affd640 3 client.4485.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:44.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.473+0000 7fec3affd640 20 client.4485.objecter dump_active .. 0 homeless 2024-03-29T10:42:44.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.477+0000 7fec4a59c640 10 monclient: get_auth_request con 0x7fec2c06e390 auth_method 0 2024-03-29T10:42:44.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.477+0000 7fec3affd640 1 -- 172.21.15.84:0/114701333 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fec4004e720 con 0x7fec4c0b0860 2024-03-29T10:42:44.460 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.477+0000 7fec3affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:44.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.477+0000 7fec3affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:44.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.481+0000 7fec3affd640 10 client.4485.objecter ms_handle_connect 0x7fec2c06e390 2024-03-29T10:42:44.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.800+0000 7fec50833640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:44.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.800+0000 7fec50833640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:44.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.800+0000 7fec50833640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:42:44.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.800+0000 7fec50833640 1 -- 172.21.15.84:0/114701333 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec4c075f10 con 0x7fec4c0b0860 2024-03-29T10:42:44.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.802+0000 7fec3affd640 1 -- 172.21.15.84:0/114701333 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7fec4004e720 con 0x7fec4c0b0860 2024-03-29T10:42:44.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.802+0000 7fec3affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:44.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.802+0000 7fec3affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:44.786 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:42:44.786 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:44.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.805+0000 7fec50833640 10 librados: watch_flush enter 2024-03-29T10:42:44.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.805+0000 7fec50833640 10 librados: watch_flush exit 2024-03-29T10:42:44.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.805+0000 7fec50833640 20 client.4485.objecter shutdown clearing up homeless session... 2024-03-29T10:42:44.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.805+0000 7fec50833640 10 client.4485.objecter successfully canceled tick 2024-03-29T10:42:44.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.805+0000 7fec50833640 1 -- 172.21.15.84:0/114701333 >> v1:172.21.15.86:6816/3337225337 conn(0x7fec2c06e390 legacy=0x7fec2c070850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:44.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.805+0000 7fec50833640 10 monclient: shutdown 2024-03-29T10:42:44.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.805+0000 7fec50833640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:44.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.805+0000 7fec50833640 1 -- 172.21.15.84:0/114701333 >> v1:172.21.15.84:6789/0 conn(0x7fec4c0b0860 legacy=0x7fec4c1daa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:44.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.806+0000 7fec50833640 1 -- 172.21.15.84:0/114701333 shutdown_connections 2024-03-29T10:42:44.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.806+0000 7fec50833640 1 -- 172.21.15.84:0/114701333 >> 172.21.15.84:0/114701333 conn(0x7fec4c06b660 msgr2=0x7fec4c13e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:44.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.806+0000 7fec50833640 1 -- 172.21.15.84:0/114701333 shutdown_connections 2024-03-29T10:42:44.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.806+0000 7fec50833640 1 -- 172.21.15.84:0/114701333 wait complete. 2024-03-29T10:42:44.790 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:44.806+0000 7fec50833640 1 librados: shutdown 2024-03-29T10:42:44.805 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:51.612 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:51.761 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.778+0000 7fa505925640 10 monclient: get_monmap_and_config 2024-03-29T10:42:51.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.778+0000 7fa505925640 10 monclient: build_initial_monmap 2024-03-29T10:42:51.762 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.778+0000 7fa505925640 10 monclient: monmap: 2024-03-29T10:42:51.762 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:42:51.762 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:51.762 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:42:51.779710+0000 2024-03-29T10:42:51.762 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:42:51.779710+0000 2024-03-29T10:42:51.762 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:51.762 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:51.762 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:42:51.762 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:42:51.762 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:42:51.763 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:51.764 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.781+0000 7fa505925640 1 Processor -- start 2024-03-29T10:42:51.764 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.781+0000 7fa505925640 1 -- start start 2024-03-29T10:42:51.764 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.781+0000 7fa505925640 10 monclient: init 2024-03-29T10:42:51.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.782+0000 7fa505925640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:51.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.782+0000 7fa505925640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:42:51.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.782+0000 7fa505925640 10 monclient(hunting): picked mon.noname-a con 0x7fa5001457f0 addr v1:172.21.15.84:6789/0 2024-03-29T10:42:51.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.782+0000 7fa505925640 10 monclient(hunting): picked mon.noname-c con 0x7fa50013f650 addr v1:172.21.15.86:6789/0 2024-03-29T10:42:51.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.782+0000 7fa505925640 10 monclient(hunting): picked mon.noname-b con 0x7fa5000a9d70 addr v1:172.21.15.84:6790/0 2024-03-29T10:42:51.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.782+0000 7fa505925640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5000aaef0 con 0x7fa5001457f0 2024-03-29T10:42:51.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.782+0000 7fa505925640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa500145bd0 con 0x7fa50013f650 2024-03-29T10:42:51.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.782+0000 7fa505925640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa500145eb0 con 0x7fa5000a9d70 2024-03-29T10:42:51.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.782+0000 7fa505925640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:51.765 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.782+0000 7fa505925640 10 monclient(hunting): authenticate will time out at 1274.045346s 2024-03-29T10:42:51.767 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.783+0000 7fa4ff7fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fa5000a9d70 0x7fa50014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:55200/0 (socket says 172.21.15.84:55200) 2024-03-29T10:42:51.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.783+0000 7fa4ff7fe640 1 -- 172.21.15.84:0/2340883405 learned_addr learned my addr 172.21.15.84:0/2340883405 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:42:51.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.783+0000 7fa4fdffb640 1 -- 172.21.15.84:0/2340883405 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 259647888 0 0) 0x7fa500145eb0 con 0x7fa5000a9d70 2024-03-29T10:42:51.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.783+0000 7fa4fdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:42:51.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.783+0000 7fa4fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:51.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.783+0000 7fa4fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:51.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.783+0000 7fa4fdffb640 10 monclient(hunting): my global_id is 4574 2024-03-29T10:42:51.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.783+0000 7fa4fdffb640 1 -- 172.21.15.84:0/2340883405 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4e8003520 con 0x7fa5000a9d70 2024-03-29T10:42:51.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.784+0000 7fa4fdffb640 1 -- 172.21.15.84:0/2340883405 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1490378274 0 0) 0x7fa4e8003520 con 0x7fa5000a9d70 2024-03-29T10:42:51.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.784+0000 7fa4fdffb640 1 -- 172.21.15.84:0/2340883405 >> v1:172.21.15.86:6789/0 conn(0x7fa50013f650 legacy=0x7fa5000b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:51.768 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.784+0000 7fa4fdffb640 1 -- 172.21.15.84:0/2340883405 >> v1:172.21.15.84:6789/0 conn(0x7fa5001457f0 legacy=0x7fa50013ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:51.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.785+0000 7fa4fdffb640 10 monclient: _finish_hunting 0 2024-03-29T10:42:51.771 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.785+0000 7fa4fdffb640 1 monclient: found mon.noname-b 2024-03-29T10:42:51.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.785+0000 7fa4fdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:51.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.785+0000 7fa4fdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:42:51.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.785+0000 7fa4fdffb640 1 -- 172.21.15.84:0/2340883405 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa500146340 con 0x7fa5000a9d70 2024-03-29T10:42:51.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.785+0000 7fa4fdffb640 10 monclient: _finish_auth 0 2024-03-29T10:42:51.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.785+0000 7fa4fdffb640 10 monclient: _check_auth_tickets 2024-03-29T10:42:51.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.785+0000 7fa4fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:51.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.786+0000 7fa4fdffb640 1 -- 172.21.15.84:0/2340883405 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa4f8001a10 con 0x7fa5000a9d70 2024-03-29T10:42:51.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.786+0000 7fa505925640 5 monclient: authenticate success, global_id 4574 2024-03-29T10:42:51.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.786+0000 7fa505925640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:51.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.786+0000 7fa4fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:51.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.786+0000 7fa4fdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:51.772 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.786+0000 7fa4fdffb640 10 monclient: dump: 2024-03-29T10:42:51.773 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:51.773 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:51.773 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:51.773 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:51.773 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:51.773 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:51.773 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:51.773 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:51.773 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:51.773 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:51.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.786+0000 7fa505925640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:51.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.786+0000 7fa4fdffb640 1 -- 172.21.15.84:0/2340883405 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4f8001ca0 con 0x7fa5000a9d70 2024-03-29T10:42:51.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.786+0000 7fa4fdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:51.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.786+0000 7fa505925640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:51.773 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.786+0000 7fa505925640 4 set_mon_vals no callback set 2024-03-29T10:42:51.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.786+0000 7fa4fdffb640 1 -- 172.21.15.84:0/2340883405 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa4f8002030 con 0x7fa5000a9d70 2024-03-29T10:42:51.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.787+0000 7fa4fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:51.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.787+0000 7fa4fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:51.774 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.787+0000 7fa4fdffb640 10 monclient: dump: 2024-03-29T10:42:51.774 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:51.774 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:51.774 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:51.774 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:51.774 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:51.774 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:51.774 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:51.774 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:51.774 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:51.775 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:51.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.787+0000 7fa505925640 10 monclient: shutdown 2024-03-29T10:42:51.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.787+0000 7fa505925640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:51.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.787+0000 7fa505925640 1 -- 172.21.15.84:0/2340883405 >> v1:172.21.15.84:6790/0 conn(0x7fa5000a9d70 legacy=0x7fa50014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:51.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.788+0000 7fa505925640 1 -- 172.21.15.84:0/2340883405 shutdown_connections 2024-03-29T10:42:51.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.788+0000 7fa505925640 1 -- 172.21.15.84:0/2340883405 >> 172.21.15.84:0/2340883405 conn(0x7fa500066a00 msgr2=0x7fa500066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:51.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.788+0000 7fa505925640 1 -- 172.21.15.84:0/2340883405 shutdown_connections 2024-03-29T10:42:51.775 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.788+0000 7fa505925640 1 -- 172.21.15.84:0/2340883405 wait complete. 2024-03-29T10:42:51.778 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.789+0000 7fa505925640 10 monclient: build_initial_monmap 2024-03-29T10:42:51.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.789+0000 7fa505925640 10 monclient: monmap: 2024-03-29T10:42:51.779 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:42:51.779 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:51.779 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:42:51.779 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:42:51.779 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:51.779 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:51.779 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:42:51.779 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:42:51.779 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:42:51.779 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:51.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.789+0000 7fa505925640 1 librados: starting msgr at 2024-03-29T10:42:51.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.789+0000 7fa505925640 1 librados: starting objecter 2024-03-29T10:42:51.779 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.789+0000 7fa505925640 1 Processor -- start 2024-03-29T10:42:51.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.790+0000 7fa505925640 1 -- start start 2024-03-29T10:42:51.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.790+0000 7fa505925640 1 librados: setting wanted keys 2024-03-29T10:42:51.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.790+0000 7fa505925640 1 librados: calling monclient init 2024-03-29T10:42:51.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.790+0000 7fa505925640 10 monclient: init 2024-03-29T10:42:51.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.790+0000 7fa505925640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:51.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.791+0000 7fa505925640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:42:51.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.791+0000 7fa505925640 10 monclient(hunting): picked mon.noname-a con 0x7fa5000a9d70 addr v1:172.21.15.84:6789/0 2024-03-29T10:42:51.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.791+0000 7fa505925640 10 monclient(hunting): picked mon.noname-c con 0x7fa50013f650 addr v1:172.21.15.84:6790/0 2024-03-29T10:42:51.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.791+0000 7fa505925640 10 monclient(hunting): picked mon.noname-b con 0x7fa5001457f0 addr v1:172.21.15.86:6789/0 2024-03-29T10:42:51.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.791+0000 7fa505925640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa500071f60 con 0x7fa5000a9d70 2024-03-29T10:42:51.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.791+0000 7fa505925640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa50006ea60 con 0x7fa5001457f0 2024-03-29T10:42:51.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.791+0000 7fa505925640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa50006ed40 con 0x7fa50013f650 2024-03-29T10:42:51.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.791+0000 7fa505925640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:51.780 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.791+0000 7fa505925640 10 monclient(hunting): authenticate will time out at 1274.054267s 2024-03-29T10:42:51.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.791+0000 7fa4fe7fc640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fa50013f650 0x7fa5001423f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:56702/0 (socket says 172.21.15.84:56702) 2024-03-29T10:42:51.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.791+0000 7fa4fe7fc640 1 -- 172.21.15.84:0/1265380369 learned_addr learned my addr 172.21.15.84:0/1265380369 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:42:51.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.792+0000 7fa4e7fff640 10 client.?.objecter ms_handle_connect 0x7fa50013f650 2024-03-29T10:42:51.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.792+0000 7fa4e7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:51.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.792+0000 7fa4e7fff640 1 -- 172.21.15.84:0/1265380369 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930882314 0 0) 0x7fa50006ed40 con 0x7fa50013f650 2024-03-29T10:42:51.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.792+0000 7fa4e7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:42:51.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.792+0000 7fa4e7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:51.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.792+0000 7fa4e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:51.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.792+0000 7fa4e7fff640 10 monclient(hunting): my global_id is 4577 2024-03-29T10:42:51.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.792+0000 7fa4e7fff640 1 -- 172.21.15.84:0/1265380369 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4ec004eb0 con 0x7fa50013f650 2024-03-29T10:42:51.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.793+0000 7fa4e7fff640 1 -- 172.21.15.84:0/1265380369 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2917403839 0 0) 0x7fa4ec004eb0 con 0x7fa50013f650 2024-03-29T10:42:51.781 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.793+0000 7fa4e7fff640 1 -- 172.21.15.84:0/1265380369 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa50006ed40 con 0x7fa50013f650 2024-03-29T10:42:51.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.793+0000 7fa4e7fff640 1 -- 172.21.15.84:0/1265380369 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa4f4001870 con 0x7fa50013f650 2024-03-29T10:42:51.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.793+0000 7fa4e7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:51.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.793+0000 7fa4e7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:42:51.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.794+0000 7fa4e7fff640 10 monclient(hunting): dump: 2024-03-29T10:42:51.782 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:51.782 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:51.782 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:51.782 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:51.782 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:51.782 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:51.782 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:51.782 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:51.782 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:51.783 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:51.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.794+0000 7fa4e7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:51.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.794+0000 7fa4e7fff640 1 -- 172.21.15.84:0/1265380369 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1076704312 0 0) 0x7fa50006ed40 con 0x7fa50013f650 2024-03-29T10:42:51.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.794+0000 7fa4e7fff640 1 -- 172.21.15.84:0/1265380369 >> v1:172.21.15.86:6789/0 conn(0x7fa5001457f0 legacy=0x7fa500144810 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:42:51.783 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.794+0000 7fa4e7fff640 1 -- 172.21.15.84:0/1265380369 >> v1:172.21.15.84:6789/0 conn(0x7fa5000a9d70 legacy=0x7fa500141c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:42:51.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.797+0000 7fa4e7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:42:51.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.797+0000 7fa4e7fff640 1 monclient: found mon.c 2024-03-29T10:42:51.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.797+0000 7fa4e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:51.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.797+0000 7fa4e7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:42:51.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.797+0000 7fa4e7fff640 1 -- 172.21.15.84:0/1265380369 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa50006f020 con 0x7fa50013f650 2024-03-29T10:42:51.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.797+0000 7fa4e7fff640 10 monclient: _finish_auth 0 2024-03-29T10:42:51.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.797+0000 7fa4e7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:42:51.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.798+0000 7fa4e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:51.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.798+0000 7fa505925640 5 monclient: authenticate success, global_id 4577 2024-03-29T10:42:51.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.798+0000 7fa505925640 10 monclient: _renew_subs 2024-03-29T10:42:51.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.798+0000 7fa505925640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:42:51.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.798+0000 7fa505925640 1 -- 172.21.15.84:0/1265380369 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa500071bf0 con 0x7fa50013f650 2024-03-29T10:42:51.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.798+0000 7fa505925640 10 client.4577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:51.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.798+0000 7fa505925640 10 monclient: _renew_subs 2024-03-29T10:42:51.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.798+0000 7fa505925640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:42:51.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.798+0000 7fa505925640 1 -- 172.21.15.84:0/1265380369 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5000710d0 con 0x7fa50013f650 2024-03-29T10:42:51.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.798+0000 7fa505925640 1 librados: init done 2024-03-29T10:42:51.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.798+0000 7fa4e7fff640 1 -- 172.21.15.84:0/1265380369 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4f40019d0 con 0x7fa50013f650 2024-03-29T10:42:51.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.798+0000 7fa4e7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:51.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.798+0000 7fa4e7fff640 1 -- 172.21.15.84:0/1265380369 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa4f4004aa0 con 0x7fa50013f650 2024-03-29T10:42:51.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.798+0000 7fa4e7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:51.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.798+0000 7fa4e7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:51.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.798+0000 7fa4e7fff640 10 monclient: dump: 2024-03-29T10:42:51.788 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:51.788 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:51.788 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:51.788 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:51.788 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:51.788 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:51.788 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:51.788 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:51.788 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:51.789 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:51.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.798+0000 7fa4fd7fa640 4 set_mon_vals no callback set 2024-03-29T10:42:51.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.800+0000 7fa4e7fff640 1 -- 172.21.15.84:0/1265380369 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7fa4f401aa70 con 0x7fa50013f650 2024-03-29T10:42:51.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.801+0000 7fa4e7fff640 1 -- 172.21.15.84:0/1265380369 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7fa4f4084a70 con 0x7fa50013f650 2024-03-29T10:42:51.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.801+0000 7fa4e7fff640 10 client.4577.objecter ms_dispatch 0x7fa500000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:51.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.801+0000 7fa4e7fff640 3 client.4577.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:51.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.801+0000 7fa4e7fff640 3 client.4577.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:51.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.801+0000 7fa4feffd640 10 monclient: get_auth_request con 0x7fa4ec06dd30 auth_method 0 2024-03-29T10:42:51.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.801+0000 7fa4e7fff640 20 client.4577.objecter dump_active .. 0 homeless 2024-03-29T10:42:51.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.802+0000 7fa505925640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:51.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.802+0000 7fa505925640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:51.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.802+0000 7fa505925640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:42:51.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.802+0000 7fa505925640 1 -- 172.21.15.84:0/1265380369 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4e0005080 con 0x7fa50013f650 2024-03-29T10:42:51.789 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.802+0000 7fa4e7fff640 10 client.4577.objecter ms_handle_connect 0x7fa4ec06dd30 2024-03-29T10:42:51.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.810+0000 7fa4e7fff640 1 -- 172.21.15.84:0/1265380369 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa4f4050350 con 0x7fa50013f650 2024-03-29T10:42:51.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.810+0000 7fa4e7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:51.793 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:51.810+0000 7fa4e7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:52.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:52.142+0000 7fa505925640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:52.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:52.142+0000 7fa505925640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:52.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:52.142+0000 7fa505925640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:42:52.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:52.142+0000 7fa505925640 1 -- 172.21.15.84:0/1265380369 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4e0005330 con 0x7fa50013f650 2024-03-29T10:42:52.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:52.145+0000 7fa4e7fff640 1 -- 172.21.15.84:0/1265380369 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7fa4f4052280 con 0x7fa50013f650 2024-03-29T10:42:52.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:52.145+0000 7fa4e7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:52.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:52.145+0000 7fa4e7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:52.129 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:42:52.129 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:52.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:52.149+0000 7fa505925640 10 librados: watch_flush enter 2024-03-29T10:42:52.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:52.149+0000 7fa505925640 10 librados: watch_flush exit 2024-03-29T10:42:52.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:52.149+0000 7fa505925640 20 client.4577.objecter shutdown clearing up homeless session... 2024-03-29T10:42:52.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:52.149+0000 7fa505925640 10 client.4577.objecter successfully canceled tick 2024-03-29T10:42:52.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:52.149+0000 7fa505925640 1 -- 172.21.15.84:0/1265380369 >> v1:172.21.15.86:6816/3337225337 conn(0x7fa4ec06dd30 legacy=0x7fa4ec0701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:52.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:52.149+0000 7fa505925640 10 monclient: shutdown 2024-03-29T10:42:52.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:52.149+0000 7fa505925640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:52.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:52.149+0000 7fa505925640 1 -- 172.21.15.84:0/1265380369 >> v1:172.21.15.84:6790/0 conn(0x7fa50013f650 legacy=0x7fa5001423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:52.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:52.149+0000 7fa505925640 1 -- 172.21.15.84:0/1265380369 shutdown_connections 2024-03-29T10:42:52.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:52.149+0000 7fa505925640 1 -- 172.21.15.84:0/1265380369 >> 172.21.15.84:0/1265380369 conn(0x7fa500066a00 msgr2=0x7fa50013aaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:52.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:52.152+0000 7fa505925640 1 -- 172.21.15.84:0/1265380369 shutdown_connections 2024-03-29T10:42:52.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:52.152+0000 7fa505925640 1 -- 172.21.15.84:0/1265380369 wait complete. 2024-03-29T10:42:52.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:52.152+0000 7fa505925640 1 librados: shutdown 2024-03-29T10:42:52.153 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:42:58.961 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:42:59.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.123+0000 7fc38459d640 10 monclient: get_monmap_and_config 2024-03-29T10:42:59.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.124+0000 7fc38459d640 10 monclient: build_initial_monmap 2024-03-29T10:42:59.111 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.124+0000 7fc38459d640 10 monclient: monmap: 2024-03-29T10:42:59.111 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:42:59.111 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:59.111 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:42:59.124974+0000 2024-03-29T10:42:59.112 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:42:59.124974+0000 2024-03-29T10:42:59.112 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:59.112 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:59.112 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:42:59.112 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:42:59.112 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:42:59.112 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:59.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.124+0000 7fc38459d640 1 Processor -- start 2024-03-29T10:42:59.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.125+0000 7fc38459d640 1 -- start start 2024-03-29T10:42:59.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.125+0000 7fc38459d640 10 monclient: init 2024-03-29T10:42:59.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.126+0000 7fc38459d640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:59.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.126+0000 7fc38459d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:42:59.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.126+0000 7fc38459d640 10 monclient(hunting): picked mon.noname-a con 0x7fc37c0ad6c0 addr v1:172.21.15.84:6789/0 2024-03-29T10:42:59.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.126+0000 7fc38459d640 10 monclient(hunting): picked mon.noname-c con 0x7fc37c0af6c0 addr v1:172.21.15.86:6789/0 2024-03-29T10:42:59.112 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.126+0000 7fc38459d640 10 monclient(hunting): picked mon.noname-b con 0x7fc37c14fdc0 addr v1:172.21.15.84:6790/0 2024-03-29T10:42:59.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.126+0000 7fc38459d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc37c0a94e0 con 0x7fc37c0ad6c0 2024-03-29T10:42:59.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.126+0000 7fc38459d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc37c0af3a0 con 0x7fc37c0af6c0 2024-03-29T10:42:59.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.126+0000 7fc38459d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc37c0afb60 con 0x7fc37c14fdc0 2024-03-29T10:42:59.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.126+0000 7fc38459d640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:59.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.126+0000 7fc38459d640 10 monclient(hunting): authenticate will time out at 1281.389297s 2024-03-29T10:42:59.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.126+0000 7fc382b3a640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fc37c14fdc0 0x7fc37c15a8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:34482/0 (socket says 172.21.15.84:34482) 2024-03-29T10:42:59.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.126+0000 7fc382b3a640 1 -- 172.21.15.84:0/3609016739 learned_addr learned my addr 172.21.15.84:0/3609016739 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:42:59.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.127+0000 7fc381337640 1 -- 172.21.15.84:0/3609016739 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 74670520 0 0) 0x7fc37c0afb60 con 0x7fc37c14fdc0 2024-03-29T10:42:59.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.127+0000 7fc381337640 10 monclient(hunting): _negotiate 2024-03-29T10:42:59.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.127+0000 7fc381337640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:59.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.127+0000 7fc381337640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:59.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.127+0000 7fc381337640 10 monclient(hunting): my global_id is 4580 2024-03-29T10:42:59.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.127+0000 7fc381337640 1 -- 172.21.15.84:0/3609016739 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc368003500 con 0x7fc37c14fdc0 2024-03-29T10:42:59.113 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.127+0000 7fc381337640 1 -- 172.21.15.84:0/3609016739 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 57019613 0 0) 0x7fc368003500 con 0x7fc37c14fdc0 2024-03-29T10:42:59.114 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.127+0000 7fc381337640 1 -- 172.21.15.84:0/3609016739 >> v1:172.21.15.86:6789/0 conn(0x7fc37c0af6c0 legacy=0x7fc37c152980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:42:59.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.130+0000 7fc381337640 1 -- 172.21.15.84:0/3609016739 >> v1:172.21.15.84:6789/0 conn(0x7fc37c0ad6c0 legacy=0x7fc37c14f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:59.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.130+0000 7fc381337640 10 monclient: _finish_hunting 0 2024-03-29T10:42:59.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.130+0000 7fc381337640 1 monclient: found mon.noname-b 2024-03-29T10:42:59.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.130+0000 7fc381337640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:59.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.130+0000 7fc381337640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:42:59.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.130+0000 7fc381337640 1 -- 172.21.15.84:0/3609016739 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc37c0aab80 con 0x7fc37c14fdc0 2024-03-29T10:42:59.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.130+0000 7fc381337640 10 monclient: _finish_auth 0 2024-03-29T10:42:59.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.130+0000 7fc381337640 10 monclient: _check_auth_tickets 2024-03-29T10:42:59.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.130+0000 7fc381337640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:59.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.130+0000 7fc38459d640 5 monclient: authenticate success, global_id 4580 2024-03-29T10:42:59.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.130+0000 7fc38459d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:59.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.130+0000 7fc381337640 1 -- 172.21.15.84:0/3609016739 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fc374001a10 con 0x7fc37c14fdc0 2024-03-29T10:42:59.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.130+0000 7fc381337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:59.116 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.130+0000 7fc381337640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:59.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.130+0000 7fc381337640 10 monclient: dump: 2024-03-29T10:42:59.117 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:59.117 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:59.117 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:59.117 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:59.117 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:59.117 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:59.117 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:59.117 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:59.117 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:59.117 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:59.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.131+0000 7fc38459d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:42:59.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.131+0000 7fc381337640 1 -- 172.21.15.84:0/3609016739 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc374001be0 con 0x7fc37c14fdc0 2024-03-29T10:42:59.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.131+0000 7fc381337640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:59.117 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.131+0000 7fc38459d640 10 monclient: get_monmap_and_config success 2024-03-29T10:42:59.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.131+0000 7fc38459d640 4 set_mon_vals no callback set 2024-03-29T10:42:59.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.131+0000 7fc381337640 1 -- 172.21.15.84:0/3609016739 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fc374002030 con 0x7fc37c14fdc0 2024-03-29T10:42:59.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.131+0000 7fc381337640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:59.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.131+0000 7fc381337640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:42:59.118 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.131+0000 7fc381337640 10 monclient: dump: 2024-03-29T10:42:59.118 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:59.118 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:59.118 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:59.118 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:59.118 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:59.118 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:59.118 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:59.118 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:59.118 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:59.118 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:59.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.132+0000 7fc38459d640 10 monclient: shutdown 2024-03-29T10:42:59.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.132+0000 7fc38459d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:59.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.132+0000 7fc38459d640 1 -- 172.21.15.84:0/3609016739 >> v1:172.21.15.84:6790/0 conn(0x7fc37c14fdc0 legacy=0x7fc37c15a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:59.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.133+0000 7fc38459d640 1 -- 172.21.15.84:0/3609016739 shutdown_connections 2024-03-29T10:42:59.119 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.133+0000 7fc38459d640 1 -- 172.21.15.84:0/3609016739 >> 172.21.15.84:0/3609016739 conn(0x7fc37c0b35e0 msgr2=0x7fc37c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:59.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.134+0000 7fc38459d640 1 -- 172.21.15.84:0/3609016739 shutdown_connections 2024-03-29T10:42:59.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.134+0000 7fc38459d640 1 -- 172.21.15.84:0/3609016739 wait complete. 2024-03-29T10:42:59.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.135+0000 7fc38459d640 10 monclient: build_initial_monmap 2024-03-29T10:42:59.120 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.135+0000 7fc38459d640 10 monclient: monmap: 2024-03-29T10:42:59.120 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:42:59.120 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:42:59.120 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:42:59.120 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:42:59.120 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:42:59.120 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:59.120 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:42:59.120 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:42:59.121 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:42:59.121 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:59.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.136+0000 7fc38459d640 1 librados: starting msgr at 2024-03-29T10:42:59.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.136+0000 7fc38459d640 1 librados: starting objecter 2024-03-29T10:42:59.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.136+0000 7fc38459d640 1 Processor -- start 2024-03-29T10:42:59.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.136+0000 7fc38459d640 1 -- start start 2024-03-29T10:42:59.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.136+0000 7fc38459d640 1 librados: setting wanted keys 2024-03-29T10:42:59.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.136+0000 7fc38459d640 1 librados: calling monclient init 2024-03-29T10:42:59.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.136+0000 7fc38459d640 10 monclient: init 2024-03-29T10:42:59.121 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.137+0000 7fc38459d640 10 monclient: _reopen_session rank -1 2024-03-29T10:42:59.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.137+0000 7fc38459d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:42:59.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.137+0000 7fc38459d640 10 monclient(hunting): picked mon.noname-c con 0x7fc37c0ad6c0 addr v1:172.21.15.84:6790/0 2024-03-29T10:42:59.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.137+0000 7fc38459d640 10 monclient(hunting): picked mon.noname-a con 0x7fc37c0af6c0 addr v1:172.21.15.84:6789/0 2024-03-29T10:42:59.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.137+0000 7fc38459d640 10 monclient(hunting): picked mon.noname-b con 0x7fc37c14fdc0 addr v1:172.21.15.86:6789/0 2024-03-29T10:42:59.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.137+0000 7fc38459d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc37c1ec750 con 0x7fc37c0af6c0 2024-03-29T10:42:59.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.137+0000 7fc38459d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc37c14d960 con 0x7fc37c14fdc0 2024-03-29T10:42:59.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.137+0000 7fc38459d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc37c152100 con 0x7fc37c0ad6c0 2024-03-29T10:42:59.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.137+0000 7fc38459d640 10 monclient(hunting): _renew_subs 2024-03-29T10:42:59.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.137+0000 7fc38459d640 10 monclient(hunting): authenticate will time out at 1281.400550s 2024-03-29T10:42:59.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.140+0000 7fc382b3a640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc37c14fdc0 0x7fc37c1f6d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:58403/0 (socket says 172.21.15.84:58403) 2024-03-29T10:42:59.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.140+0000 7fc382b3a640 1 -- 172.21.15.84:0/1743969924 learned_addr learned my addr 172.21.15.84:0/1743969924 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:42:59.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.141+0000 7fc372ffd640 10 client.?.objecter ms_handle_connect 0x7fc37c14fdc0 2024-03-29T10:42:59.124 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.141+0000 7fc372ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:59.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.141+0000 7fc372ffd640 1 -- 172.21.15.84:0/1743969924 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291463143 0 0) 0x7fc37c14d960 con 0x7fc37c14fdc0 2024-03-29T10:42:59.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.141+0000 7fc372ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:59.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.141+0000 7fc372ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:59.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.141+0000 7fc372ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:59.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.141+0000 7fc372ffd640 10 monclient(hunting): my global_id is 4459 2024-03-29T10:42:59.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.141+0000 7fc372ffd640 1 -- 172.21.15.84:0/1743969924 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc360004eb0 con 0x7fc37c14fdc0 2024-03-29T10:42:59.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.142+0000 7fc372ffd640 10 client.?.objecter ms_handle_connect 0x7fc37c0ad6c0 2024-03-29T10:42:59.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.143+0000 7fc372ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:59.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.143+0000 7fc372ffd640 10 client.?.objecter ms_handle_connect 0x7fc37c0af6c0 2024-03-29T10:42:59.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.143+0000 7fc372ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:42:59.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.143+0000 7fc372ffd640 1 -- 172.21.15.84:0/1743969924 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1910038400 0 0) 0x7fc360004eb0 con 0x7fc37c14fdc0 2024-03-29T10:42:59.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.143+0000 7fc372ffd640 1 -- 172.21.15.84:0/1743969924 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc37c14d960 con 0x7fc37c14fdc0 2024-03-29T10:42:59.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.143+0000 7fc372ffd640 1 -- 172.21.15.84:0/1743969924 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3893803442 0 0) 0x7fc37c1ec750 con 0x7fc37c0af6c0 2024-03-29T10:42:59.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.143+0000 7fc372ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:42:59.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.143+0000 7fc372ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:42:59.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.143+0000 7fc372ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:42:59.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.143+0000 7fc372ffd640 10 monclient(hunting): my global_id is 4491 2024-03-29T10:42:59.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.143+0000 7fc372ffd640 1 -- 172.21.15.84:0/1743969924 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc360004eb0 con 0x7fc37c0af6c0 2024-03-29T10:42:59.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.143+0000 7fc372ffd640 1 -- 172.21.15.84:0/1743969924 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fc374001f30 con 0x7fc37c14fdc0 2024-03-29T10:42:59.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.143+0000 7fc372ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:59.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.143+0000 7fc372ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:42:59.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.143+0000 7fc372ffd640 10 monclient(hunting): dump: 2024-03-29T10:42:59.130 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:59.130 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:59.131 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:59.131 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:59.131 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:59.131 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:59.131 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:59.131 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:59.131 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:59.131 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:59.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.143+0000 7fc372ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:42:59.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.143+0000 7fc372ffd640 1 -- 172.21.15.84:0/1743969924 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1132700434 0 0) 0x7fc360004eb0 con 0x7fc37c0af6c0 2024-03-29T10:42:59.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.144+0000 7fc372ffd640 1 -- 172.21.15.84:0/1743969924 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc37c1ec750 con 0x7fc37c0af6c0 2024-03-29T10:42:59.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.144+0000 7fc372ffd640 1 -- 172.21.15.84:0/1743969924 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2189411720 0 0) 0x7fc37c14d960 con 0x7fc37c14fdc0 2024-03-29T10:42:59.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.144+0000 7fc372ffd640 1 -- 172.21.15.84:0/1743969924 >> v1:172.21.15.84:6790/0 conn(0x7fc37c0ad6c0 legacy=0x7fc37c1519f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:59.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.144+0000 7fc372ffd640 1 -- 172.21.15.84:0/1743969924 >> v1:172.21.15.84:6789/0 conn(0x7fc37c0af6c0 legacy=0x7fc37c1ecbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:59.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.144+0000 7fc372ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:42:59.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.144+0000 7fc372ffd640 1 monclient: found mon.b 2024-03-29T10:42:59.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.144+0000 7fc372ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:42:59.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.144+0000 7fc372ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:42:59.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.144+0000 7fc372ffd640 1 -- 172.21.15.84:0/1743969924 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc37c1ed2f0 con 0x7fc37c14fdc0 2024-03-29T10:42:59.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.144+0000 7fc372ffd640 10 monclient: _finish_auth 0 2024-03-29T10:42:59.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.144+0000 7fc372ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:42:59.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.144+0000 7fc372ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:42:59.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.145+0000 7fc38459d640 5 monclient: authenticate success, global_id 4459 2024-03-29T10:42:59.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.145+0000 7fc38459d640 10 monclient: _renew_subs 2024-03-29T10:42:59.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.145+0000 7fc38459d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:42:59.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.145+0000 7fc38459d640 1 -- 172.21.15.84:0/1743969924 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc37c14f070 con 0x7fc37c14fdc0 2024-03-29T10:42:59.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.145+0000 7fc38459d640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:42:59.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.145+0000 7fc38459d640 10 monclient: _renew_subs 2024-03-29T10:42:59.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.145+0000 7fc38459d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:42:59.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.145+0000 7fc38459d640 1 -- 172.21.15.84:0/1743969924 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc37c150f50 con 0x7fc37c14fdc0 2024-03-29T10:42:59.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.145+0000 7fc38459d640 1 librados: init done 2024-03-29T10:42:59.134 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.147+0000 7fc38459d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:59.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.147+0000 7fc38459d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:59.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.147+0000 7fc38459d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:42:59.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.147+0000 7fc372ffd640 1 -- 172.21.15.84:0/1743969924 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc374002090 con 0x7fc37c14fdc0 2024-03-29T10:42:59.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.147+0000 7fc38459d640 1 -- 172.21.15.84:0/1743969924 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc37c075f10 con 0x7fc37c14fdc0 2024-03-29T10:42:59.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.147+0000 7fc372ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:42:59.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.147+0000 7fc372ffd640 1 -- 172.21.15.84:0/1743969924 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fc374004fb0 con 0x7fc37c14fdc0 2024-03-29T10:42:59.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.147+0000 7fc372ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:42:59.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.147+0000 7fc373fff640 4 set_mon_vals no callback set 2024-03-29T10:42:59.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.148+0000 7fc372ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:42:59.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.148+0000 7fc372ffd640 10 monclient: dump: 2024-03-29T10:42:59.135 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:42:59.135 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:42:59.135 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:59.135 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:42:59.136 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:42:59.136 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:42:59.136 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:42:59.136 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:42:59.136 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:42:59.136 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:42:59.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.150+0000 7fc372ffd640 1 -- 172.21.15.84:0/1743969924 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7fc374005190 con 0x7fc37c14fdc0 2024-03-29T10:42:59.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.151+0000 7fc372ffd640 1 -- 172.21.15.84:0/1743969924 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7fc374084e60 con 0x7fc37c14fdc0 2024-03-29T10:42:59.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.151+0000 7fc372ffd640 10 client.4459.objecter ms_dispatch 0x7fc37c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:42:59.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.151+0000 7fc372ffd640 3 client.4459.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:42:59.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.151+0000 7fc372ffd640 3 client.4459.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:42:59.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.151+0000 7fc372ffd640 20 client.4459.objecter dump_active .. 0 homeless 2024-03-29T10:42:59.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.152+0000 7fc382339640 10 monclient: get_auth_request con 0x7fc36006dfb0 auth_method 0 2024-03-29T10:42:59.137 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.152+0000 7fc372ffd640 10 client.4459.objecter ms_handle_connect 0x7fc36006dfb0 2024-03-29T10:42:59.141 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.158+0000 7fc372ffd640 1 -- 172.21.15.84:0/1743969924 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc374050740 con 0x7fc37c14fdc0 2024-03-29T10:42:59.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.158+0000 7fc372ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:42:59.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.158+0000 7fc372ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:42:59.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.482+0000 7fc38459d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:59.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.482+0000 7fc38459d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:59.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.482+0000 7fc38459d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:42:59.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.483+0000 7fc38459d640 1 -- 172.21.15.84:0/1743969924 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc37c1ec8e0 con 0x7fc37c14fdc0 2024-03-29T10:42:59.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.484+0000 7fc372ffd640 1 -- 172.21.15.84:0/1743969924 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7fc374052670 con 0x7fc37c14fdc0 2024-03-29T10:42:59.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.484+0000 7fc372ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:42:59.467 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.484+0000 7fc372ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:42:59.467 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:42:59.467 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:42:59.470 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.487+0000 7fc38459d640 10 librados: watch_flush enter 2024-03-29T10:42:59.471 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.488+0000 7fc38459d640 10 librados: watch_flush exit 2024-03-29T10:42:59.471 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.488+0000 7fc38459d640 20 client.4459.objecter shutdown clearing up homeless session... 2024-03-29T10:42:59.471 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.488+0000 7fc38459d640 10 client.4459.objecter successfully canceled tick 2024-03-29T10:42:59.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.492+0000 7fc38459d640 1 -- 172.21.15.84:0/1743969924 >> v1:172.21.15.86:6816/3337225337 conn(0x7fc36006dfb0 legacy=0x7fc360070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:59.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.492+0000 7fc38459d640 10 monclient: shutdown 2024-03-29T10:42:59.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.492+0000 7fc38459d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:42:59.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.492+0000 7fc38459d640 1 -- 172.21.15.84:0/1743969924 >> v1:172.21.15.86:6789/0 conn(0x7fc37c14fdc0 legacy=0x7fc37c1f6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:42:59.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.493+0000 7fc38459d640 1 -- 172.21.15.84:0/1743969924 shutdown_connections 2024-03-29T10:42:59.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.493+0000 7fc38459d640 1 -- 172.21.15.84:0/1743969924 >> 172.21.15.84:0/1743969924 conn(0x7fc37c0b35e0 msgr2=0x7fc37c08a950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:42:59.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.493+0000 7fc38459d640 1 -- 172.21.15.84:0/1743969924 shutdown_connections 2024-03-29T10:42:59.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.493+0000 7fc38459d640 1 -- 172.21.15.84:0/1743969924 wait complete. 2024-03-29T10:42:59.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:42:59.493+0000 7fc38459d640 1 librados: shutdown 2024-03-29T10:42:59.492 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:06.299 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:06.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.467+0000 7f4927b0f640 10 monclient: get_monmap_and_config 2024-03-29T10:43:06.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.467+0000 7f4927b0f640 10 monclient: build_initial_monmap 2024-03-29T10:43:06.451 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.467+0000 7f4927b0f640 10 monclient: monmap: 2024-03-29T10:43:06.451 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:43:06.451 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:06.451 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:43:06.468769+0000 2024-03-29T10:43:06.451 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:43:06.468769+0000 2024-03-29T10:43:06.451 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:06.451 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:06.451 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:43:06.452 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:43:06.452 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:43:06.452 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:06.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.468+0000 7f4927b0f640 1 Processor -- start 2024-03-29T10:43:06.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.468+0000 7f4927b0f640 1 -- start start 2024-03-29T10:43:06.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.468+0000 7f4927b0f640 10 monclient: init 2024-03-29T10:43:06.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.469+0000 7f4927b0f640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:06.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.469+0000 7f4927b0f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:43:06.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.469+0000 7f4927b0f640 10 monclient(hunting): picked mon.noname-a con 0x7f49201422b0 addr v1:172.21.15.84:6789/0 2024-03-29T10:43:06.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.469+0000 7f4927b0f640 10 monclient(hunting): picked mon.noname-c con 0x7f4920142690 addr v1:172.21.15.86:6789/0 2024-03-29T10:43:06.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.469+0000 7f4927b0f640 10 monclient(hunting): picked mon.noname-b con 0x7f49200b17d0 addr v1:172.21.15.84:6790/0 2024-03-29T10:43:06.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.469+0000 7f4927b0f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4920142c80 con 0x7f49201422b0 2024-03-29T10:43:06.453 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.469+0000 7f4927b0f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49200aae40 con 0x7f4920142690 2024-03-29T10:43:06.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.469+0000 7f4927b0f640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492013fbd0 con 0x7f49200b17d0 2024-03-29T10:43:06.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.469+0000 7f4927b0f640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:06.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.469+0000 7f4927b0f640 10 monclient(hunting): authenticate will time out at 1288.733025s 2024-03-29T10:43:06.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.470+0000 7f49260ac640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f49200b17d0 0x7f49200b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:33808/0 (socket says 172.21.15.84:33808) 2024-03-29T10:43:06.454 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.470+0000 7f49260ac640 1 -- 172.21.15.84:0/628362202 learned_addr learned my addr 172.21.15.84:0/628362202 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:43:06.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.471+0000 7f49248a9640 1 -- 172.21.15.84:0/628362202 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 688264450 0 0) 0x7f492013fbd0 con 0x7f49200b17d0 2024-03-29T10:43:06.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.471+0000 7f49248a9640 10 monclient(hunting): _negotiate 2024-03-29T10:43:06.455 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.471+0000 7f49248a9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:06.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.471+0000 7f49248a9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:06.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.471+0000 7f49248a9640 10 monclient(hunting): my global_id is 4586 2024-03-29T10:43:06.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.471+0000 7f49248a9640 1 -- 172.21.15.84:0/628362202 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f490c003520 con 0x7f49200b17d0 2024-03-29T10:43:06.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.472+0000 7f49248a9640 1 -- 172.21.15.84:0/628362202 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4204402942 0 0) 0x7f490c003520 con 0x7f49200b17d0 2024-03-29T10:43:06.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.472+0000 7f49248a9640 1 -- 172.21.15.84:0/628362202 >> v1:172.21.15.86:6789/0 conn(0x7f4920142690 legacy=0x7f49200b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:43:06.456 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.472+0000 7f49248a9640 1 -- 172.21.15.84:0/628362202 >> v1:172.21.15.84:6789/0 conn(0x7f49201422b0 legacy=0x7f492013f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:43:06.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.474+0000 7f49248a9640 10 monclient: _finish_hunting 0 2024-03-29T10:43:06.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.474+0000 7f49248a9640 1 monclient: found mon.noname-b 2024-03-29T10:43:06.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.474+0000 7f49248a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:06.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.474+0000 7f49248a9640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:43:06.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.474+0000 7f49248a9640 1 -- 172.21.15.84:0/628362202 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4920151940 con 0x7f49200b17d0 2024-03-29T10:43:06.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.474+0000 7f49248a9640 10 monclient: _finish_auth 0 2024-03-29T10:43:06.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.474+0000 7f49248a9640 10 monclient: _check_auth_tickets 2024-03-29T10:43:06.461 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.474+0000 7f49248a9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:06.462 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.474+0000 7f49248a9640 1 -- 172.21.15.84:0/628362202 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4918001a10 con 0x7f49200b17d0 2024-03-29T10:43:06.462 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.474+0000 7f49248a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:06.462 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.474+0000 7f49248a9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:06.462 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.474+0000 7f49248a9640 10 monclient: dump: 2024-03-29T10:43:06.462 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:06.462 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:06.462 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:06.463 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:06.463 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:06.463 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:06.463 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:06.463 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:06.463 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:06.463 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:06.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.475+0000 7f4927b0f640 5 monclient: authenticate success, global_id 4586 2024-03-29T10:43:06.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.475+0000 7f4927b0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:06.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.475+0000 7f49248a9640 1 -- 172.21.15.84:0/628362202 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4918001b70 con 0x7f49200b17d0 2024-03-29T10:43:06.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.475+0000 7f49248a9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:06.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.475+0000 7f49248a9640 1 -- 172.21.15.84:0/628362202 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4918001fc0 con 0x7f49200b17d0 2024-03-29T10:43:06.463 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.475+0000 7f49248a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:06.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.475+0000 7f49248a9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:06.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.475+0000 7f49248a9640 10 monclient: dump: 2024-03-29T10:43:06.464 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:06.464 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:06.464 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:06.464 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:06.464 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:06.464 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:06.464 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:06.464 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:06.464 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:06.464 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:06.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.475+0000 7f4927b0f640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:06.464 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.475+0000 7f4927b0f640 4 set_mon_vals no callback set 2024-03-29T10:43:06.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.476+0000 7f4927b0f640 10 monclient: shutdown 2024-03-29T10:43:06.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.476+0000 7f4927b0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:06.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.476+0000 7f4927b0f640 1 -- 172.21.15.84:0/628362202 >> v1:172.21.15.84:6790/0 conn(0x7f49200b17d0 legacy=0x7f49200b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:06.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.477+0000 7f4927b0f640 1 -- 172.21.15.84:0/628362202 shutdown_connections 2024-03-29T10:43:06.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.477+0000 7f4927b0f640 1 -- 172.21.15.84:0/628362202 >> 172.21.15.84:0/628362202 conn(0x7f4920066a00 msgr2=0x7f4920066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:06.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.478+0000 7f4927b0f640 1 -- 172.21.15.84:0/628362202 shutdown_connections 2024-03-29T10:43:06.465 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.478+0000 7f4927b0f640 1 -- 172.21.15.84:0/628362202 wait complete. 2024-03-29T10:43:06.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.479+0000 7f4927b0f640 10 monclient: build_initial_monmap 2024-03-29T10:43:06.466 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.479+0000 7f4927b0f640 10 monclient: monmap: 2024-03-29T10:43:06.466 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:43:06.466 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:06.466 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:43:06.466 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:43:06.466 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:06.466 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:06.466 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:43:06.466 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:43:06.466 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:43:06.466 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:06.471 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.479+0000 7f4927b0f640 1 librados: starting msgr at 2024-03-29T10:43:06.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.479+0000 7f4927b0f640 1 librados: starting objecter 2024-03-29T10:43:06.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.479+0000 7f4927b0f640 1 Processor -- start 2024-03-29T10:43:06.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.480+0000 7f4927b0f640 1 -- start start 2024-03-29T10:43:06.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.480+0000 7f4927b0f640 1 librados: setting wanted keys 2024-03-29T10:43:06.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.480+0000 7f4927b0f640 1 librados: calling monclient init 2024-03-29T10:43:06.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.480+0000 7f4927b0f640 10 monclient: init 2024-03-29T10:43:06.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.480+0000 7f4927b0f640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:06.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.480+0000 7f4927b0f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:43:06.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.480+0000 7f4927b0f640 10 monclient(hunting): picked mon.noname-a con 0x7f49200b17d0 addr v1:172.21.15.84:6789/0 2024-03-29T10:43:06.472 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.481+0000 7f4927b0f640 10 monclient(hunting): picked mon.noname-b con 0x7f49201422b0 addr v1:172.21.15.86:6789/0 2024-03-29T10:43:06.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.481+0000 7f4927b0f640 10 monclient(hunting): picked mon.noname-c con 0x7f4920142690 addr v1:172.21.15.84:6790/0 2024-03-29T10:43:06.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.481+0000 7f4927b0f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4920131f80 con 0x7f49200b17d0 2024-03-29T10:43:06.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.481+0000 7f4927b0f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4920132b20 con 0x7f49201422b0 2024-03-29T10:43:06.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.481+0000 7f4927b0f640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f492012f680 con 0x7f4920142690 2024-03-29T10:43:06.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.481+0000 7f4927b0f640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:06.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.481+0000 7f4927b0f640 10 monclient(hunting): authenticate will time out at 1288.744719s 2024-03-29T10:43:06.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.482+0000 7f49260ac640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f4920142690 0x7f492012ef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:35036/0 (socket says 172.21.15.84:35036) 2024-03-29T10:43:06.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.482+0000 7f49260ac640 1 -- 172.21.15.84:0/3406295415 learned_addr learned my addr 172.21.15.84:0/3406295415 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:43:06.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.482+0000 7f49167fc640 10 client.?.objecter ms_handle_connect 0x7f4920142690 2024-03-29T10:43:06.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.482+0000 7f49167fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:06.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.482+0000 7f49167fc640 1 -- 172.21.15.84:0/3406295415 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3465138497 0 0) 0x7f492012f680 con 0x7f4920142690 2024-03-29T10:43:06.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.485+0000 7f49167fc640 10 monclient(hunting): _negotiate 2024-03-29T10:43:06.473 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.485+0000 7f49167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:06.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.485+0000 7f49167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:06.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.485+0000 7f49167fc640 10 monclient(hunting): my global_id is 4589 2024-03-29T10:43:06.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.485+0000 7f49167fc640 1 -- 172.21.15.84:0/3406295415 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4904004eb0 con 0x7f4920142690 2024-03-29T10:43:06.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.486+0000 7f49167fc640 10 client.?.objecter ms_handle_connect 0x7f49201422b0 2024-03-29T10:43:06.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.486+0000 7f49167fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:06.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.486+0000 7f49167fc640 10 client.?.objecter ms_handle_connect 0x7f49200b17d0 2024-03-29T10:43:06.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.486+0000 7f49167fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:06.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.486+0000 7f49167fc640 1 -- 172.21.15.84:0/3406295415 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 590497045 0 0) 0x7f4904004eb0 con 0x7f4920142690 2024-03-29T10:43:06.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.486+0000 7f49167fc640 1 -- 172.21.15.84:0/3406295415 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f492012f680 con 0x7f4920142690 2024-03-29T10:43:06.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.486+0000 7f49167fc640 1 -- 172.21.15.84:0/3406295415 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4918001e80 con 0x7f4920142690 2024-03-29T10:43:06.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.486+0000 7f49167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:06.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.486+0000 7f49167fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:06.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.486+0000 7f49167fc640 10 monclient(hunting): dump: 2024-03-29T10:43:06.475 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:06.475 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:06.475 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:06.475 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:06.475 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:06.475 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:06.475 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:06.476 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:06.476 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:06.476 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:06.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.486+0000 7f49167fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:06.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.486+0000 7f49167fc640 1 -- 172.21.15.84:0/3406295415 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042334665 0 0) 0x7f4920132b20 con 0x7f49201422b0 2024-03-29T10:43:06.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.486+0000 7f49167fc640 10 monclient(hunting): _negotiate 2024-03-29T10:43:06.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.486+0000 7f49167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:06.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.486+0000 7f49167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:06.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.486+0000 7f49167fc640 10 monclient(hunting): my global_id is 4462 2024-03-29T10:43:06.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.486+0000 7f49167fc640 1 -- 172.21.15.84:0/3406295415 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4904004eb0 con 0x7f49201422b0 2024-03-29T10:43:06.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.486+0000 7f49167fc640 1 -- 172.21.15.84:0/3406295415 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3209658789 0 0) 0x7f4920131f80 con 0x7f49200b17d0 2024-03-29T10:43:06.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.486+0000 7f49167fc640 10 monclient(hunting): _negotiate 2024-03-29T10:43:06.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.486+0000 7f49167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:06.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.486+0000 7f49167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:06.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.486+0000 7f49167fc640 10 monclient(hunting): my global_id is 4494 2024-03-29T10:43:06.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.486+0000 7f49167fc640 1 -- 172.21.15.84:0/3406295415 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4920132b20 con 0x7f49200b17d0 2024-03-29T10:43:06.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.487+0000 7f49167fc640 1 -- 172.21.15.84:0/3406295415 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2438292235 0 0) 0x7f492012f680 con 0x7f4920142690 2024-03-29T10:43:06.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.487+0000 7f49167fc640 1 -- 172.21.15.84:0/3406295415 >> v1:172.21.15.86:6789/0 conn(0x7f49201422b0 legacy=0x7f492012e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:06.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.487+0000 7f49167fc640 1 -- 172.21.15.84:0/3406295415 >> v1:172.21.15.84:6789/0 conn(0x7f49200b17d0 legacy=0x7f4920132410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:06.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.487+0000 7f49167fc640 10 monclient: _finish_hunting 0 2024-03-29T10:43:06.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.487+0000 7f49167fc640 1 monclient: found mon.c 2024-03-29T10:43:06.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.487+0000 7f49167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:06.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.487+0000 7f49167fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:43:06.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.487+0000 7f49167fc640 1 -- 172.21.15.84:0/3406295415 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492006ec60 con 0x7f4920142690 2024-03-29T10:43:06.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.487+0000 7f49167fc640 10 monclient: _finish_auth 0 2024-03-29T10:43:06.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.487+0000 7f49167fc640 10 monclient: _check_auth_tickets 2024-03-29T10:43:06.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.487+0000 7f49167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:06.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.487+0000 7f4927b0f640 5 monclient: authenticate success, global_id 4589 2024-03-29T10:43:06.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.487+0000 7f4927b0f640 10 monclient: _renew_subs 2024-03-29T10:43:06.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.487+0000 7f4927b0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:43:06.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.487+0000 7f4927b0f640 1 -- 172.21.15.84:0/3406295415 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f492006ef40 con 0x7f4920142690 2024-03-29T10:43:06.478 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.488+0000 7f4927b0f640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:06.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.488+0000 7f4927b0f640 10 monclient: _renew_subs 2024-03-29T10:43:06.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.488+0000 7f4927b0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:43:06.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.488+0000 7f4927b0f640 1 -- 172.21.15.84:0/3406295415 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f492013bc60 con 0x7f4920142690 2024-03-29T10:43:06.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.488+0000 7f4927b0f640 1 librados: init done 2024-03-29T10:43:06.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.489+0000 7f490bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:06.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.489+0000 7f490bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:06.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.489+0000 7f490bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:43:06.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.489+0000 7f490bfff640 1 -- 172.21.15.84:0/3406295415 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4920130da0 con 0x7f4920142690 2024-03-29T10:43:06.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.489+0000 7f49167fc640 1 -- 172.21.15.84:0/3406295415 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4918001fe0 con 0x7f4920142690 2024-03-29T10:43:06.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.489+0000 7f49167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:06.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.489+0000 7f49167fc640 1 -- 172.21.15.84:0/3406295415 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4918005110 con 0x7f4920142690 2024-03-29T10:43:06.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.489+0000 7f49167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:06.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.489+0000 7f49167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:06.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.489+0000 7f49167fc640 10 monclient: dump: 2024-03-29T10:43:06.481 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:06.481 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:06.481 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:06.482 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:06.482 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:06.482 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:06.482 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:06.482 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:06.482 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:06.482 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:06.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.489+0000 7f49177fe640 4 set_mon_vals no callback set 2024-03-29T10:43:06.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.491+0000 7f49167fc640 1 -- 172.21.15.84:0/3406295415 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f491801b0e0 con 0x7f4920142690 2024-03-29T10:43:06.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.492+0000 7f49167fc640 1 -- 172.21.15.84:0/3406295415 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f4918084f90 con 0x7f4920142690 2024-03-29T10:43:06.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.492+0000 7f49167fc640 10 client.4589.objecter ms_dispatch 0x7f4920000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:06.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.492+0000 7f49167fc640 3 client.4589.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:06.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.492+0000 7f49167fc640 3 client.4589.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:06.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.492+0000 7f49167fc640 20 client.4589.objecter dump_active .. 0 homeless 2024-03-29T10:43:06.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.493+0000 7f49258ab640 10 monclient: get_auth_request con 0x7f490406de90 auth_method 0 2024-03-29T10:43:06.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.493+0000 7f49167fc640 10 client.4589.objecter ms_handle_connect 0x7f490406de90 2024-03-29T10:43:06.484 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.498+0000 7f49167fc640 1 -- 172.21.15.84:0/3406295415 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f49180b28b0 con 0x7f4920142690 2024-03-29T10:43:06.484 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.498+0000 7f49167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:06.484 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.498+0000 7f49167fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:06.799 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.816+0000 7f490bfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:06.799 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.816+0000 7f490bfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:06.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.816+0000 7f490bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:43:06.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.816+0000 7f490bfff640 1 -- 172.21.15.84:0/3406295415 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4920075f10 con 0x7f4920142690 2024-03-29T10:43:06.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.817+0000 7f49167fc640 1 -- 172.21.15.84:0/3406295415 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f49180506a0 con 0x7f4920142690 2024-03-29T10:43:06.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.817+0000 7f49167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:06.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.817+0000 7f49167fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:06.801 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:43:06.802 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:06.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.820+0000 7f490bfff640 10 librados: watch_flush enter 2024-03-29T10:43:06.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.820+0000 7f490bfff640 10 librados: watch_flush exit 2024-03-29T10:43:06.804 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.820+0000 7f490bfff640 20 client.4589.objecter shutdown clearing up homeless session... 2024-03-29T10:43:06.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.820+0000 7f490bfff640 10 client.4589.objecter successfully canceled tick 2024-03-29T10:43:06.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.822+0000 7f490bfff640 1 -- 172.21.15.84:0/3406295415 >> v1:172.21.15.86:6816/3337225337 conn(0x7f490406de90 legacy=0x7f4904070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:06.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.822+0000 7f490bfff640 10 monclient: shutdown 2024-03-29T10:43:06.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.822+0000 7f490bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:06.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.822+0000 7f490bfff640 1 -- 172.21.15.84:0/3406295415 >> v1:172.21.15.84:6790/0 conn(0x7f4920142690 legacy=0x7f492012ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:06.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.822+0000 7f490bfff640 1 -- 172.21.15.84:0/3406295415 shutdown_connections 2024-03-29T10:43:06.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.822+0000 7f490bfff640 1 -- 172.21.15.84:0/3406295415 >> 172.21.15.84:0/3406295415 conn(0x7f4920066a00 msgr2=0x7f49200b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:06.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.822+0000 7f490bfff640 1 -- 172.21.15.84:0/3406295415 shutdown_connections 2024-03-29T10:43:06.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.822+0000 7f490bfff640 1 -- 172.21.15.84:0/3406295415 wait complete. 2024-03-29T10:43:06.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:06.823+0000 7f490bfff640 1 librados: shutdown 2024-03-29T10:43:06.826 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:13.634 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:13.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.798+0000 7f4f3ee25640 10 monclient: get_monmap_and_config 2024-03-29T10:43:13.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.799+0000 7f4f3ee25640 10 monclient: build_initial_monmap 2024-03-29T10:43:13.782 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.799+0000 7f4f3ee25640 10 monclient: monmap: 2024-03-29T10:43:13.782 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:43:13.782 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:13.782 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:43:13.799978+0000 2024-03-29T10:43:13.783 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:43:13.799978+0000 2024-03-29T10:43:13.783 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:13.783 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:13.783 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:43:13.783 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:43:13.783 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:43:13.783 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:13.784 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.801+0000 7f4f3ee25640 1 Processor -- start 2024-03-29T10:43:13.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.801+0000 7f4f3ee25640 1 -- start start 2024-03-29T10:43:13.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.801+0000 7f4f3ee25640 10 monclient: init 2024-03-29T10:43:13.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.802+0000 7f4f3ee25640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:13.785 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.802+0000 7f4f3ee25640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:43:13.786 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.802+0000 7f4f3ee25640 10 monclient(hunting): picked mon.noname-a con 0x7f4f381457f0 addr v1:172.21.15.84:6789/0 2024-03-29T10:43:13.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.803+0000 7f4f3ee25640 10 monclient(hunting): picked mon.noname-b con 0x7f4f3813f650 addr v1:172.21.15.84:6790/0 2024-03-29T10:43:13.787 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.804+0000 7f4f3ee25640 10 monclient(hunting): picked mon.noname-c con 0x7f4f380a9d70 addr v1:172.21.15.86:6789/0 2024-03-29T10:43:13.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.804+0000 7f4f3ee25640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f380aaef0 con 0x7f4f381457f0 2024-03-29T10:43:13.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.804+0000 7f4f3ee25640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f38145bd0 con 0x7f4f380a9d70 2024-03-29T10:43:13.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.804+0000 7f4f3ee25640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f38145eb0 con 0x7f4f3813f650 2024-03-29T10:43:13.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.804+0000 7f4f3ee25640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:13.788 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.804+0000 7f4f3ee25640 10 monclient(hunting): authenticate will time out at 1296.067482s 2024-03-29T10:43:13.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.808+0000 7f4f3cbc1640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f4f381457f0 0x7f4f3813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:45591/0 (socket says 172.21.15.84:45591) 2024-03-29T10:43:13.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.808+0000 7f4f3cbc1640 1 -- 172.21.15.84:0/2381121267 learned_addr learned my addr 172.21.15.84:0/2381121267 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:43:13.797 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.809+0000 7f4f377fe640 1 -- 172.21.15.84:0/2381121267 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2487979176 0 0) 0x7f4f38145bd0 con 0x7f4f380a9d70 2024-03-29T10:43:13.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.809+0000 7f4f377fe640 10 monclient(hunting): _negotiate 2024-03-29T10:43:13.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.809+0000 7f4f377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:13.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.809+0000 7f4f377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:13.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.809+0000 7f4f377fe640 10 monclient(hunting): my global_id is 4465 2024-03-29T10:43:13.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.809+0000 7f4f377fe640 1 -- 172.21.15.84:0/2381121267 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f28003520 con 0x7f4f380a9d70 2024-03-29T10:43:13.798 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.809+0000 7f4f377fe640 1 -- 172.21.15.84:0/2381121267 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4132761478 0 0) 0x7f4f38145eb0 con 0x7f4f3813f650 2024-03-29T10:43:13.799 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.809+0000 7f4f377fe640 10 monclient(hunting): _negotiate 2024-03-29T10:43:13.799 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.810+0000 7f4f377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:13.799 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.810+0000 7f4f377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:13.799 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.810+0000 7f4f377fe640 10 monclient(hunting): my global_id is 4592 2024-03-29T10:43:13.799 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.810+0000 7f4f377fe640 1 -- 172.21.15.84:0/2381121267 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f38145bd0 con 0x7f4f3813f650 2024-03-29T10:43:13.799 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.810+0000 7f4f377fe640 1 -- 172.21.15.84:0/2381121267 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1302042184 0 0) 0x7f4f28003520 con 0x7f4f380a9d70 2024-03-29T10:43:13.799 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.810+0000 7f4f377fe640 1 -- 172.21.15.84:0/2381121267 >> v1:172.21.15.84:6790/0 conn(0x7f4f3813f650 legacy=0x7f4f380b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:13.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.810+0000 7f4f377fe640 1 -- 172.21.15.84:0/2381121267 >> v1:172.21.15.84:6789/0 conn(0x7f4f381457f0 legacy=0x7f4f3813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:13.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.810+0000 7f4f377fe640 10 monclient: _finish_hunting 0 2024-03-29T10:43:13.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.810+0000 7f4f377fe640 1 monclient: found mon.noname-c 2024-03-29T10:43:13.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.811+0000 7f4f377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:13.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.811+0000 7f4f377fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-29T10:43:13.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.811+0000 7f4f377fe640 1 -- 172.21.15.84:0/2381121267 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f38146340 con 0x7f4f380a9d70 2024-03-29T10:43:13.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.811+0000 7f4f377fe640 10 monclient: _finish_auth 0 2024-03-29T10:43:13.800 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.811+0000 7f4f377fe640 10 monclient: _check_auth_tickets 2024-03-29T10:43:13.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.811+0000 7f4f377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:13.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.811+0000 7f4f377fe640 1 -- 172.21.15.84:0/2381121267 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4f30001a10 con 0x7f4f380a9d70 2024-03-29T10:43:13.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.811+0000 7f4f377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:13.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.811+0000 7f4f377fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:13.801 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.811+0000 7f4f377fe640 10 monclient: dump: 2024-03-29T10:43:13.801 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:13.801 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:13.801 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:13.801 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:13.801 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:13.801 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:13.801 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:13.801 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:13.801 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:13.802 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:13.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.811+0000 7f4f377fe640 1 -- 172.21.15.84:0/2381121267 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f30001c30 con 0x7f4f380a9d70 2024-03-29T10:43:13.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.811+0000 7f4f377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:13.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.811+0000 7f4f377fe640 1 -- 172.21.15.84:0/2381121267 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4f30001fc0 con 0x7f4f380a9d70 2024-03-29T10:43:13.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.811+0000 7f4f377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:13.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.811+0000 7f4f377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:13.802 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.811+0000 7f4f377fe640 10 monclient: dump: 2024-03-29T10:43:13.802 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:13.802 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:13.802 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:13.802 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:13.802 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:13.802 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:13.802 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:13.803 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:13.803 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:13.803 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:13.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.812+0000 7f4f3ee25640 5 monclient: authenticate success, global_id 4465 2024-03-29T10:43:13.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.812+0000 7f4f3ee25640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:13.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.812+0000 7f4f3ee25640 4 set_mon_vals no callback set 2024-03-29T10:43:13.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.813+0000 7f4f3ee25640 10 monclient: shutdown 2024-03-29T10:43:13.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.813+0000 7f4f3ee25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:13.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.813+0000 7f4f3ee25640 1 -- 172.21.15.84:0/2381121267 >> v1:172.21.15.86:6789/0 conn(0x7f4f380a9d70 legacy=0x7f4f3814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:13.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.814+0000 7f4f3ee25640 1 -- 172.21.15.84:0/2381121267 shutdown_connections 2024-03-29T10:43:13.803 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.814+0000 7f4f3ee25640 1 -- 172.21.15.84:0/2381121267 >> 172.21.15.84:0/2381121267 conn(0x7f4f38066a00 msgr2=0x7f4f38066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:13.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.814+0000 7f4f3ee25640 1 -- 172.21.15.84:0/2381121267 shutdown_connections 2024-03-29T10:43:13.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.814+0000 7f4f3ee25640 1 -- 172.21.15.84:0/2381121267 wait complete. 2024-03-29T10:43:13.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.815+0000 7f4f3ee25640 10 monclient: build_initial_monmap 2024-03-29T10:43:13.805 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.815+0000 7f4f3ee25640 10 monclient: monmap: 2024-03-29T10:43:13.805 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:43:13.805 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:13.805 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:43:13.805 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:43:13.805 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:13.805 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:13.806 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:43:13.806 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:43:13.806 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:43:13.806 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:13.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.816+0000 7f4f3ee25640 1 librados: starting msgr at 2024-03-29T10:43:13.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.816+0000 7f4f3ee25640 1 librados: starting objecter 2024-03-29T10:43:13.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.816+0000 7f4f3ee25640 1 Processor -- start 2024-03-29T10:43:13.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.816+0000 7f4f3ee25640 1 -- start start 2024-03-29T10:43:13.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.816+0000 7f4f3ee25640 1 librados: setting wanted keys 2024-03-29T10:43:13.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.816+0000 7f4f3ee25640 1 librados: calling monclient init 2024-03-29T10:43:13.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.816+0000 7f4f3ee25640 10 monclient: init 2024-03-29T10:43:13.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.817+0000 7f4f3ee25640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:13.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.817+0000 7f4f3ee25640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:43:13.806 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.817+0000 7f4f3ee25640 10 monclient(hunting): picked mon.noname-c con 0x7f4f380a9d70 addr v1:172.21.15.84:6790/0 2024-03-29T10:43:13.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.819+0000 7f4f3ee25640 10 monclient(hunting): picked mon.noname-a con 0x7f4f3813f650 addr v1:172.21.15.84:6789/0 2024-03-29T10:43:13.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.819+0000 7f4f3ee25640 10 monclient(hunting): picked mon.noname-b con 0x7f4f381457f0 addr v1:172.21.15.86:6789/0 2024-03-29T10:43:13.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.819+0000 7f4f3ee25640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f38133130 con 0x7f4f3813f650 2024-03-29T10:43:13.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.819+0000 7f4f3ee25640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f38133410 con 0x7f4f381457f0 2024-03-29T10:43:13.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.819+0000 7f4f3ee25640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f381322f0 con 0x7f4f380a9d70 2024-03-29T10:43:13.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.819+0000 7f4f3ee25640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:13.807 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.819+0000 7f4f3ee25640 10 monclient(hunting): authenticate will time out at 1296.083070s 2024-03-29T10:43:13.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.823+0000 7f4f37fff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f4f3813f650 0x7f4f3813fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:47731/0 (socket says 172.21.15.84:47731) 2024-03-29T10:43:13.808 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.823+0000 7f4f37fff640 1 -- 172.21.15.84:0/4016148273 learned_addr learned my addr 172.21.15.84:0/4016148273 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:43:13.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.828+0000 7f4f357fa640 10 client.?.objecter ms_handle_connect 0x7f4f3813f650 2024-03-29T10:43:13.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.828+0000 7f4f357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:13.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.828+0000 7f4f357fa640 10 client.?.objecter ms_handle_connect 0x7f4f380a9d70 2024-03-29T10:43:13.811 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.828+0000 7f4f357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:13.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.828+0000 7f4f357fa640 1 -- 172.21.15.84:0/4016148273 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 632624940 0 0) 0x7f4f38133130 con 0x7f4f3813f650 2024-03-29T10:43:13.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.829+0000 7f4f357fa640 10 monclient(hunting): _negotiate 2024-03-29T10:43:13.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.829+0000 7f4f357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:13.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.829+0000 7f4f357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:13.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.829+0000 7f4f357fa640 10 monclient(hunting): my global_id is 4500 2024-03-29T10:43:13.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.829+0000 7f4f357fa640 1 -- 172.21.15.84:0/4016148273 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f1c004eb0 con 0x7f4f3813f650 2024-03-29T10:43:13.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.829+0000 7f4f357fa640 1 -- 172.21.15.84:0/4016148273 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3929612211 0 0) 0x7f4f381322f0 con 0x7f4f380a9d70 2024-03-29T10:43:13.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.829+0000 7f4f357fa640 10 monclient(hunting): _negotiate 2024-03-29T10:43:13.815 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.829+0000 7f4f357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:13.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.829+0000 7f4f357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:13.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.829+0000 7f4f357fa640 10 monclient(hunting): my global_id is 4595 2024-03-29T10:43:13.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.829+0000 7f4f357fa640 1 -- 172.21.15.84:0/4016148273 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f38133130 con 0x7f4f380a9d70 2024-03-29T10:43:13.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.830+0000 7f4f357fa640 1 -- 172.21.15.84:0/4016148273 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2261210533 0 0) 0x7f4f38133130 con 0x7f4f380a9d70 2024-03-29T10:43:13.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.830+0000 7f4f357fa640 1 -- 172.21.15.84:0/4016148273 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f381322f0 con 0x7f4f380a9d70 2024-03-29T10:43:13.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.830+0000 7f4f357fa640 1 -- 172.21.15.84:0/4016148273 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4f24001780 con 0x7f4f380a9d70 2024-03-29T10:43:13.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.830+0000 7f4f357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:13.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.830+0000 7f4f357fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:13.816 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.830+0000 7f4f357fa640 10 monclient(hunting): dump: 2024-03-29T10:43:13.816 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:13.816 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:13.816 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:13.816 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:13.816 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:13.817 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:13.817 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:13.817 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:13.817 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:13.817 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:13.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.830+0000 7f4f357fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:13.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.830+0000 7f4f357fa640 1 -- 172.21.15.84:0/4016148273 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 934235805 0 0) 0x7f4f381322f0 con 0x7f4f380a9d70 2024-03-29T10:43:13.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.830+0000 7f4f357fa640 1 -- 172.21.15.84:0/4016148273 >> v1:172.21.15.86:6789/0 conn(0x7f4f381457f0 legacy=0x7f4f381e2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:13.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.830+0000 7f4f357fa640 1 -- 172.21.15.84:0/4016148273 >> v1:172.21.15.84:6789/0 conn(0x7f4f3813f650 legacy=0x7f4f3813fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:13.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.830+0000 7f4f357fa640 10 monclient: _finish_hunting 0 2024-03-29T10:43:13.817 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.830+0000 7f4f357fa640 1 monclient: found mon.c 2024-03-29T10:43:13.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.830+0000 7f4f357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:13.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.831+0000 7f4f357fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:43:13.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.831+0000 7f4f357fa640 1 -- 172.21.15.84:0/4016148273 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f381421b0 con 0x7f4f380a9d70 2024-03-29T10:43:13.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.831+0000 7f4f357fa640 10 monclient: _finish_auth 0 2024-03-29T10:43:13.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.831+0000 7f4f357fa640 10 monclient: _check_auth_tickets 2024-03-29T10:43:13.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.831+0000 7f4f357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:13.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.831+0000 7f4f3ee25640 5 monclient: authenticate success, global_id 4595 2024-03-29T10:43:13.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.831+0000 7f4f3ee25640 10 monclient: _renew_subs 2024-03-29T10:43:13.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.831+0000 7f4f3ee25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:43:13.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.831+0000 7f4f3ee25640 1 -- 172.21.15.84:0/4016148273 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f38131ef0 con 0x7f4f380a9d70 2024-03-29T10:43:13.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.831+0000 7f4f3ee25640 10 client.4595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:13.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.831+0000 7f4f3ee25640 10 monclient: _renew_subs 2024-03-29T10:43:13.818 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.831+0000 7f4f3ee25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:43:13.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.831+0000 7f4f3ee25640 1 -- 172.21.15.84:0/4016148273 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f38140690 con 0x7f4f380a9d70 2024-03-29T10:43:13.819 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.831+0000 7f4f3ee25640 1 librados: init done 2024-03-29T10:43:13.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.832+0000 7f4f357fa640 1 -- 172.21.15.84:0/4016148273 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f24001bb0 con 0x7f4f380a9d70 2024-03-29T10:43:13.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.832+0000 7f4f357fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:13.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.832+0000 7f4f36ffd640 4 set_mon_vals no callback set 2024-03-29T10:43:13.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.833+0000 7f4f357fa640 1 -- 172.21.15.84:0/4016148273 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4f24004980 con 0x7f4f380a9d70 2024-03-29T10:43:13.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.833+0000 7f4f357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:13.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.833+0000 7f4f357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:13.820 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.833+0000 7f4f357fa640 10 monclient: dump: 2024-03-29T10:43:13.820 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:13.820 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:13.821 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:13.821 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:13.821 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:13.821 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:13.821 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:13.821 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:13.821 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:13.821 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:13.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.833+0000 7f4f357fa640 1 -- 172.21.15.84:0/4016148273 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f4f24004ba0 con 0x7f4f380a9d70 2024-03-29T10:43:13.821 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.834+0000 7f4f22ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:13.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.834+0000 7f4f22ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:13.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.834+0000 7f4f22ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:43:13.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.834+0000 7f4f22ffd640 1 -- 172.21.15.84:0/4016148273 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f18005080 con 0x7f4f380a9d70 2024-03-29T10:43:13.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.834+0000 7f4f357fa640 1 -- 172.21.15.84:0/4016148273 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f4f24084930 con 0x7f4f380a9d70 2024-03-29T10:43:13.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.834+0000 7f4f357fa640 10 client.4595.objecter ms_dispatch 0x7f4f38000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:13.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.834+0000 7f4f357fa640 3 client.4595.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:13.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.834+0000 7f4f357fa640 3 client.4595.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:13.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.834+0000 7f4f37fff640 10 monclient: get_auth_request con 0x7f4f1c06dd70 auth_method 0 2024-03-29T10:43:13.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.834+0000 7f4f357fa640 20 client.4595.objecter dump_active .. 0 homeless 2024-03-29T10:43:13.822 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.835+0000 7f4f357fa640 10 client.4595.objecter ms_handle_connect 0x7f4f1c06dd70 2024-03-29T10:43:13.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.843+0000 7f4f357fa640 1 -- 172.21.15.84:0/4016148273 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4f240502c0 con 0x7f4f380a9d70 2024-03-29T10:43:13.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.843+0000 7f4f357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:13.826 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:13.843+0000 7f4f357fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:14.141 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:14.158+0000 7f4f22ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:14.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:14.158+0000 7f4f22ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:14.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:14.158+0000 7f4f22ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:43:14.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:14.158+0000 7f4f22ffd640 1 -- 172.21.15.84:0/4016148273 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f18005330 con 0x7f4f380a9d70 2024-03-29T10:43:14.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:14.159+0000 7f4f357fa640 1 -- 172.21.15.84:0/4016148273 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f4f240521f0 con 0x7f4f380a9d70 2024-03-29T10:43:14.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:14.159+0000 7f4f357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:14.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:14.159+0000 7f4f357fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:14.144 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:43:14.144 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:14.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:14.162+0000 7f4f22ffd640 10 librados: watch_flush enter 2024-03-29T10:43:14.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:14.162+0000 7f4f22ffd640 10 librados: watch_flush exit 2024-03-29T10:43:14.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:14.162+0000 7f4f22ffd640 20 client.4595.objecter shutdown clearing up homeless session... 2024-03-29T10:43:14.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:14.162+0000 7f4f22ffd640 10 client.4595.objecter successfully canceled tick 2024-03-29T10:43:14.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:14.162+0000 7f4f22ffd640 1 -- 172.21.15.84:0/4016148273 >> v1:172.21.15.86:6816/3337225337 conn(0x7f4f1c06dd70 legacy=0x7f4f1c070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:14.147 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:14.162+0000 7f4f22ffd640 10 monclient: shutdown 2024-03-29T10:43:14.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:14.162+0000 7f4f22ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:14.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:14.162+0000 7f4f22ffd640 1 -- 172.21.15.84:0/4016148273 >> v1:172.21.15.84:6790/0 conn(0x7f4f380a9d70 legacy=0x7f4f38142790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:14.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:14.163+0000 7f4f22ffd640 1 -- 172.21.15.84:0/4016148273 shutdown_connections 2024-03-29T10:43:14.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:14.163+0000 7f4f22ffd640 1 -- 172.21.15.84:0/4016148273 >> 172.21.15.84:0/4016148273 conn(0x7f4f38066a00 msgr2=0x7f4f380b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:14.148 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:14.163+0000 7f4f22ffd640 1 -- 172.21.15.84:0/4016148273 shutdown_connections 2024-03-29T10:43:14.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:14.163+0000 7f4f22ffd640 1 -- 172.21.15.84:0/4016148273 wait complete. 2024-03-29T10:43:14.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:14.164+0000 7f4f22ffd640 1 librados: shutdown 2024-03-29T10:43:14.162 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:20.967 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:21.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.138+0000 7f1c688fb640 10 monclient: get_monmap_and_config 2024-03-29T10:43:21.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.138+0000 7f1c688fb640 10 monclient: build_initial_monmap 2024-03-29T10:43:21.122 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.138+0000 7f1c688fb640 10 monclient: monmap: 2024-03-29T10:43:21.122 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:43:21.122 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:21.122 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:43:21.139852+0000 2024-03-29T10:43:21.122 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:43:21.139852+0000 2024-03-29T10:43:21.122 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:21.122 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:21.122 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:43:21.122 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:43:21.122 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:43:21.123 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:21.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.140+0000 7f1c688fb640 1 Processor -- start 2024-03-29T10:43:21.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.140+0000 7f1c688fb640 1 -- start start 2024-03-29T10:43:21.123 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.140+0000 7f1c688fb640 10 monclient: init 2024-03-29T10:43:21.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.140+0000 7f1c688fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:21.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.140+0000 7f1c688fb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:43:21.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.140+0000 7f1c688fb640 10 monclient(hunting): picked mon.noname-a con 0x7f1c5c007450 addr v1:172.21.15.84:6789/0 2024-03-29T10:43:21.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.140+0000 7f1c688fb640 10 monclient(hunting): picked mon.noname-c con 0x7f1c5c007830 addr v1:172.21.15.86:6789/0 2024-03-29T10:43:21.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.141+0000 7f1c688fb640 10 monclient(hunting): picked mon.noname-b con 0x7f1c5c0c5d40 addr v1:172.21.15.84:6790/0 2024-03-29T10:43:21.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.141+0000 7f1c688fb640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c5c02e0b0 con 0x7f1c5c007450 2024-03-29T10:43:21.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.141+0000 7f1c688fb640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c5c033840 con 0x7f1c5c007830 2024-03-29T10:43:21.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.141+0000 7f1c688fb640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c5c02e320 con 0x7f1c5c0c5d40 2024-03-29T10:43:21.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.141+0000 7f1c688fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:21.125 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.141+0000 7f1c688fb640 10 monclient(hunting): authenticate will time out at 1303.404145s 2024-03-29T10:43:21.126 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.144+0000 7f1c63fff640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f1c5c0c5d40 0x7f1c5c0d0f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:60764/0 (socket says 172.21.15.84:60764) 2024-03-29T10:43:21.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.144+0000 7f1c63fff640 1 -- 172.21.15.84:0/1608238297 learned_addr learned my addr 172.21.15.84:0/1608238297 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:43:21.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.144+0000 7f1c627fc640 1 -- 172.21.15.84:0/1608238297 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 956288485 0 0) 0x7f1c5c02e320 con 0x7f1c5c0c5d40 2024-03-29T10:43:21.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.144+0000 7f1c627fc640 10 monclient(hunting): _negotiate 2024-03-29T10:43:21.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.144+0000 7f1c627fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:21.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.144+0000 7f1c627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:21.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.144+0000 7f1c627fc640 10 monclient(hunting): my global_id is 4598 2024-03-29T10:43:21.127 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.144+0000 7f1c627fc640 1 -- 172.21.15.84:0/1608238297 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c54003520 con 0x7f1c5c0c5d40 2024-03-29T10:43:21.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.145+0000 7f1c627fc640 1 -- 172.21.15.84:0/1608238297 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3243413157 0 0) 0x7f1c54003520 con 0x7f1c5c0c5d40 2024-03-29T10:43:21.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.145+0000 7f1c627fc640 1 -- 172.21.15.84:0/1608238297 >> v1:172.21.15.86:6789/0 conn(0x7f1c5c007830 legacy=0x7f1c5c0c5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:43:21.128 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.145+0000 7f1c627fc640 1 -- 172.21.15.84:0/1608238297 >> v1:172.21.15.84:6789/0 conn(0x7f1c5c007450 legacy=0x7f1c5c033130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:43:21.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.146+0000 7f1c627fc640 10 monclient: _finish_hunting 0 2024-03-29T10:43:21.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.146+0000 7f1c627fc640 1 monclient: found mon.noname-b 2024-03-29T10:43:21.129 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.146+0000 7f1c627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:21.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.146+0000 7f1c627fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:43:21.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.146+0000 7f1c627fc640 1 -- 172.21.15.84:0/1608238297 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c5c02e600 con 0x7f1c5c0c5d40 2024-03-29T10:43:21.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.146+0000 7f1c627fc640 10 monclient: _finish_auth 0 2024-03-29T10:43:21.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.146+0000 7f1c627fc640 10 monclient: _check_auth_tickets 2024-03-29T10:43:21.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.146+0000 7f1c627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:21.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.146+0000 7f1c627fc640 1 -- 172.21.15.84:0/1608238297 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f1c50001690 con 0x7f1c5c0c5d40 2024-03-29T10:43:21.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.146+0000 7f1c627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:21.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.146+0000 7f1c627fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:21.130 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.146+0000 7f1c627fc640 10 monclient: dump: 2024-03-29T10:43:21.130 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:21.130 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:21.130 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:21.130 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:21.130 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:21.131 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:21.131 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:21.131 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:21.131 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:21.131 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:21.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.146+0000 7f1c688fb640 5 monclient: authenticate success, global_id 4598 2024-03-29T10:43:21.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.146+0000 7f1c688fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:21.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.148+0000 7f1c627fc640 1 -- 172.21.15.84:0/1608238297 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c500018f0 con 0x7f1c5c0c5d40 2024-03-29T10:43:21.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.148+0000 7f1c627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:21.131 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.148+0000 7f1c627fc640 1 -- 172.21.15.84:0/1608238297 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f1c50001d40 con 0x7f1c5c0c5d40 2024-03-29T10:43:21.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.148+0000 7f1c627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:21.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.148+0000 7f1c627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:21.132 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.148+0000 7f1c627fc640 10 monclient: dump: 2024-03-29T10:43:21.132 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:21.132 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:21.132 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:21.132 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:21.132 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:21.132 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:21.132 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:21.132 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:21.132 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:21.132 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:21.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.150+0000 7f1c688fb640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:21.133 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.150+0000 7f1c688fb640 4 set_mon_vals no callback set 2024-03-29T10:43:21.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.150+0000 7f1c688fb640 10 monclient: shutdown 2024-03-29T10:43:21.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.150+0000 7f1c688fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:21.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.150+0000 7f1c688fb640 1 -- 172.21.15.84:0/1608238297 >> v1:172.21.15.84:6790/0 conn(0x7f1c5c0c5d40 legacy=0x7f1c5c0d0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:21.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.151+0000 7f1c688fb640 1 -- 172.21.15.84:0/1608238297 shutdown_connections 2024-03-29T10:43:21.135 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.152+0000 7f1c688fb640 1 -- 172.21.15.84:0/1608238297 >> 172.21.15.84:0/1608238297 conn(0x7f1c5c006490 msgr2=0x7f1c5c0068a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:21.136 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.153+0000 7f1c688fb640 1 -- 172.21.15.84:0/1608238297 shutdown_connections 2024-03-29T10:43:21.137 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.153+0000 7f1c688fb640 1 -- 172.21.15.84:0/1608238297 wait complete. 2024-03-29T10:43:21.137 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.154+0000 7f1c688fb640 10 monclient: build_initial_monmap 2024-03-29T10:43:21.137 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.154+0000 7f1c688fb640 10 monclient: monmap: 2024-03-29T10:43:21.137 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:43:21.137 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:21.138 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:43:21.138 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:43:21.138 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:21.138 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:21.138 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:43:21.138 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:43:21.138 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:43:21.138 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:21.138 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.154+0000 7f1c688fb640 1 librados: starting msgr at 2024-03-29T10:43:21.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.154+0000 7f1c688fb640 1 librados: starting objecter 2024-03-29T10:43:21.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.155+0000 7f1c688fb640 1 Processor -- start 2024-03-29T10:43:21.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.155+0000 7f1c688fb640 1 -- start start 2024-03-29T10:43:21.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.155+0000 7f1c688fb640 1 librados: setting wanted keys 2024-03-29T10:43:21.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.155+0000 7f1c688fb640 1 librados: calling monclient init 2024-03-29T10:43:21.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.155+0000 7f1c688fb640 10 monclient: init 2024-03-29T10:43:21.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.156+0000 7f1c688fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:21.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.156+0000 7f1c688fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:43:21.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.156+0000 7f1c688fb640 10 monclient(hunting): picked mon.noname-c con 0x7f1c5c007450 addr v1:172.21.15.84:6790/0 2024-03-29T10:43:21.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.156+0000 7f1c688fb640 10 monclient(hunting): picked mon.noname-a con 0x7f1c5c0c5d40 addr v1:172.21.15.84:6789/0 2024-03-29T10:43:21.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.156+0000 7f1c688fb640 10 monclient(hunting): picked mon.noname-b con 0x7f1c5c0bc6e0 addr v1:172.21.15.86:6789/0 2024-03-29T10:43:21.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.156+0000 7f1c688fb640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c5c0c8550 con 0x7f1c5c0c5d40 2024-03-29T10:43:21.140 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.156+0000 7f1c688fb640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c5c0be1f0 con 0x7f1c5c0bc6e0 2024-03-29T10:43:21.141 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.156+0000 7f1c688fb640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c5c0be4d0 con 0x7f1c5c007450 2024-03-29T10:43:21.141 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.156+0000 7f1c688fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:21.141 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.156+0000 7f1c688fb640 10 monclient(hunting): authenticate will time out at 1303.419314s 2024-03-29T10:43:21.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.159+0000 7f1c63fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f1c5c0bc6e0 0x7f1c5c16b9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:43221/0 (socket says 172.21.15.84:43221) 2024-03-29T10:43:21.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.159+0000 7f1c63fff640 1 -- 172.21.15.84:0/508946553 learned_addr learned my addr 172.21.15.84:0/508946553 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:43:21.142 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.160+0000 7f1c4ffff640 10 client.?.objecter ms_handle_connect 0x7f1c5c0bc6e0 2024-03-29T10:43:21.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.160+0000 7f1c4ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:21.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.160+0000 7f1c4ffff640 10 client.?.objecter ms_handle_connect 0x7f1c5c007450 2024-03-29T10:43:21.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.160+0000 7f1c4ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:21.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.160+0000 7f1c4ffff640 1 -- 172.21.15.84:0/508946553 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941492821 0 0) 0x7f1c5c0be1f0 con 0x7f1c5c0bc6e0 2024-03-29T10:43:21.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.160+0000 7f1c4ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:43:21.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.160+0000 7f1c4ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:21.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.160+0000 7f1c4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:21.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.160+0000 7f1c4ffff640 10 monclient(hunting): my global_id is 4468 2024-03-29T10:43:21.143 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.160+0000 7f1c4ffff640 1 -- 172.21.15.84:0/508946553 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c48004eb0 con 0x7f1c5c0bc6e0 2024-03-29T10:43:21.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.160+0000 7f1c4ffff640 1 -- 172.21.15.84:0/508946553 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2972545271 0 0) 0x7f1c48004eb0 con 0x7f1c5c0bc6e0 2024-03-29T10:43:21.144 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.160+0000 7f1c4ffff640 1 -- 172.21.15.84:0/508946553 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c5c0be1f0 con 0x7f1c5c0bc6e0 2024-03-29T10:43:21.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.160+0000 7f1c4ffff640 1 -- 172.21.15.84:0/508946553 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f1c500014b0 con 0x7f1c5c0bc6e0 2024-03-29T10:43:21.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.160+0000 7f1c4ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:21.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.160+0000 7f1c4ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:21.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.160+0000 7f1c4ffff640 10 monclient(hunting): dump: 2024-03-29T10:43:21.145 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:21.145 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:21.145 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:21.145 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:21.145 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:21.145 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:21.145 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:21.145 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:21.145 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:21.145 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:21.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.160+0000 7f1c4ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:21.145 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.160+0000 7f1c4ffff640 1 -- 172.21.15.84:0/508946553 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2984072314 0 0) 0x7f1c5c0be1f0 con 0x7f1c5c0bc6e0 2024-03-29T10:43:21.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.160+0000 7f1c4ffff640 1 -- 172.21.15.84:0/508946553 >> v1:172.21.15.84:6790/0 conn(0x7f1c5c007450 legacy=0x7f1c5c0bcf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:21.146 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.160+0000 7f1c4ffff640 1 -- 172.21.15.84:0/508946553 >> v1:172.21.15.84:6789/0 conn(0x7f1c5c0c5d40 legacy=0x7f1c5c0c6e30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:43:21.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.162+0000 7f1c4ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:43:21.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.162+0000 7f1c4ffff640 1 monclient: found mon.b 2024-03-29T10:43:21.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.162+0000 7f1c4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:21.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.162+0000 7f1c4ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:43:21.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.162+0000 7f1c4ffff640 1 -- 172.21.15.84:0/508946553 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c5c0be7b0 con 0x7f1c5c0bc6e0 2024-03-29T10:43:21.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.162+0000 7f1c4ffff640 10 monclient: _finish_auth 0 2024-03-29T10:43:21.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.162+0000 7f1c4ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:43:21.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.162+0000 7f1c4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:21.149 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.162+0000 7f1c688fb640 5 monclient: authenticate success, global_id 4468 2024-03-29T10:43:21.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.162+0000 7f1c688fb640 10 monclient: _renew_subs 2024-03-29T10:43:21.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.162+0000 7f1c688fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:43:21.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.162+0000 7f1c688fb640 1 -- 172.21.15.84:0/508946553 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c5c0bd630 con 0x7f1c5c0bc6e0 2024-03-29T10:43:21.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.162+0000 7f1c688fb640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:21.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.162+0000 7f1c688fb640 10 monclient: _renew_subs 2024-03-29T10:43:21.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.162+0000 7f1c688fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:43:21.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.162+0000 7f1c688fb640 1 -- 172.21.15.84:0/508946553 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c5c16d2d0 con 0x7f1c5c0bc6e0 2024-03-29T10:43:21.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.162+0000 7f1c688fb640 1 librados: init done 2024-03-29T10:43:21.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.164+0000 7f1c688fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:21.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.164+0000 7f1c688fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:21.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.164+0000 7f1c688fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:43:21.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.164+0000 7f1c688fb640 1 -- 172.21.15.84:0/508946553 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c5c005120 con 0x7f1c5c0bc6e0 2024-03-29T10:43:21.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.164+0000 7f1c4ffff640 1 -- 172.21.15.84:0/508946553 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c50004ce0 con 0x7f1c5c0bc6e0 2024-03-29T10:43:21.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.164+0000 7f1c4ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:21.150 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.164+0000 7f1c4ffff640 1 -- 172.21.15.84:0/508946553 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f1c500050b0 con 0x7f1c5c0bc6e0 2024-03-29T10:43:21.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.164+0000 7f1c4ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:21.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.164+0000 7f1c4ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:21.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.165+0000 7f1c4ffff640 10 monclient: dump: 2024-03-29T10:43:21.151 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:21.151 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:21.151 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:21.151 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:21.151 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:21.151 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:21.151 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:21.151 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:21.151 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:21.151 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:21.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.165+0000 7f1c617fa640 4 set_mon_vals no callback set 2024-03-29T10:43:21.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.165+0000 7f1c4ffff640 1 -- 172.21.15.84:0/508946553 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f1c500052d0 con 0x7f1c5c0bc6e0 2024-03-29T10:43:21.151 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.166+0000 7f1c637fe640 10 monclient: get_auth_request con 0x7f1c4806de70 auth_method 0 2024-03-29T10:43:21.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.166+0000 7f1c4ffff640 1 -- 172.21.15.84:0/508946553 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f1c50084c90 con 0x7f1c5c0bc6e0 2024-03-29T10:43:21.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.166+0000 7f1c4ffff640 10 client.4468.objecter ms_dispatch 0x7f1c64000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:21.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.166+0000 7f1c4ffff640 3 client.4468.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:21.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.166+0000 7f1c4ffff640 3 client.4468.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:21.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.166+0000 7f1c4ffff640 20 client.4468.objecter dump_active .. 0 homeless 2024-03-29T10:43:21.152 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.166+0000 7f1c4ffff640 10 client.4468.objecter ms_handle_connect 0x7f1c4806de70 2024-03-29T10:43:21.154 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.171+0000 7f1c4ffff640 1 -- 172.21.15.84:0/508946553 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1c50050570 con 0x7f1c5c0bc6e0 2024-03-29T10:43:21.155 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.172+0000 7f1c4ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:21.155 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.172+0000 7f1c4ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:21.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.490+0000 7f1c688fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:21.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.490+0000 7f1c688fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:21.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.490+0000 7f1c688fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:43:21.474 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.490+0000 7f1c688fb640 1 -- 172.21.15.84:0/508946553 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c5c0c99d0 con 0x7f1c5c0bc6e0 2024-03-29T10:43:21.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.491+0000 7f1c4ffff640 1 -- 172.21.15.84:0/508946553 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f1c500524a0 con 0x7f1c5c0bc6e0 2024-03-29T10:43:21.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.491+0000 7f1c4ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:21.475 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.491+0000 7f1c4ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:21.475 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:43:21.475 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:21.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.495+0000 7f1c688fb640 10 librados: watch_flush enter 2024-03-29T10:43:21.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.495+0000 7f1c688fb640 10 librados: watch_flush exit 2024-03-29T10:43:21.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.495+0000 7f1c688fb640 20 client.4468.objecter shutdown clearing up homeless session... 2024-03-29T10:43:21.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.495+0000 7f1c688fb640 10 client.4468.objecter successfully canceled tick 2024-03-29T10:43:21.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.495+0000 7f1c688fb640 1 -- 172.21.15.84:0/508946553 >> v1:172.21.15.86:6816/3337225337 conn(0x7f1c4806de70 legacy=0x7f1c48070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:21.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.495+0000 7f1c688fb640 10 monclient: shutdown 2024-03-29T10:43:21.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.495+0000 7f1c688fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:21.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.495+0000 7f1c688fb640 1 -- 172.21.15.84:0/508946553 >> v1:172.21.15.86:6789/0 conn(0x7f1c5c0bc6e0 legacy=0x7f1c5c16b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:21.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.495+0000 7f1c688fb640 1 -- 172.21.15.84:0/508946553 shutdown_connections 2024-03-29T10:43:21.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.495+0000 7f1c688fb640 1 -- 172.21.15.84:0/508946553 >> 172.21.15.84:0/508946553 conn(0x7f1c5c006490 msgr2=0x7f1c5c0388b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:21.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.495+0000 7f1c688fb640 1 -- 172.21.15.84:0/508946553 shutdown_connections 2024-03-29T10:43:21.479 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.495+0000 7f1c688fb640 1 -- 172.21.15.84:0/508946553 wait complete. 2024-03-29T10:43:21.480 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:21.495+0000 7f1c688fb640 1 librados: shutdown 2024-03-29T10:43:21.498 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:28.305 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:28.476 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.493+0000 7fa091d03640 10 monclient: get_monmap_and_config 2024-03-29T10:43:28.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.493+0000 7fa091d03640 10 monclient: build_initial_monmap 2024-03-29T10:43:28.477 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.493+0000 7fa091d03640 10 monclient: monmap: 2024-03-29T10:43:28.477 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:43:28.477 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:28.477 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:43:28.494553+0000 2024-03-29T10:43:28.477 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:43:28.494553+0000 2024-03-29T10:43:28.477 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:28.477 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:28.477 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:43:28.477 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:43:28.477 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:43:28.477 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:28.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.498+0000 7fa091d03640 1 Processor -- start 2024-03-29T10:43:28.481 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.498+0000 7fa091d03640 1 -- start start 2024-03-29T10:43:28.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.498+0000 7fa091d03640 10 monclient: init 2024-03-29T10:43:28.482 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.499+0000 7fa091d03640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:28.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.499+0000 7fa091d03640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:43:28.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.499+0000 7fa091d03640 10 monclient(hunting): picked mon.noname-b con 0x7fa08c0a9750 addr v1:172.21.15.84:6790/0 2024-03-29T10:43:28.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.499+0000 7fa091d03640 10 monclient(hunting): picked mon.noname-c con 0x7fa08c0add40 addr v1:172.21.15.86:6789/0 2024-03-29T10:43:28.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.499+0000 7fa091d03640 10 monclient(hunting): picked mon.noname-a con 0x7fa08c0aad20 addr v1:172.21.15.84:6789/0 2024-03-29T10:43:28.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.499+0000 7fa091d03640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa08c149870 con 0x7fa08c0aad20 2024-03-29T10:43:28.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.499+0000 7fa091d03640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa08c0abf80 con 0x7fa08c0add40 2024-03-29T10:43:28.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.499+0000 7fa091d03640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa08c149ac0 con 0x7fa08c0a9750 2024-03-29T10:43:28.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.499+0000 7fa091d03640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:28.483 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.499+0000 7fa091d03640 10 monclient(hunting): authenticate will time out at 1310.763082s 2024-03-29T10:43:28.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.503+0000 7fa091502640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fa08c0aad20 0x7fa08c151bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:48978/0 (socket says 172.21.15.84:48978) 2024-03-29T10:43:28.486 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.503+0000 7fa091502640 1 -- 172.21.15.84:0/558606490 learned_addr learned my addr 172.21.15.84:0/558606490 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:43:28.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.504+0000 7fa08b7fe640 1 -- 172.21.15.84:0/558606490 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1292981499 0 0) 0x7fa08c149870 con 0x7fa08c0aad20 2024-03-29T10:43:28.487 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.504+0000 7fa08b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:43:28.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.504+0000 7fa08b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:28.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.504+0000 7fa08b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:28.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.504+0000 7fa08b7fe640 10 monclient(hunting): my global_id is 4503 2024-03-29T10:43:28.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.504+0000 7fa08b7fe640 1 -- 172.21.15.84:0/558606490 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa074003520 con 0x7fa08c0aad20 2024-03-29T10:43:28.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.504+0000 7fa08b7fe640 1 -- 172.21.15.84:0/558606490 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3981591348 0 0) 0x7fa08c0abf80 con 0x7fa08c0add40 2024-03-29T10:43:28.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.504+0000 7fa08b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:43:28.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.504+0000 7fa08b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:28.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.504+0000 7fa08b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:28.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.504+0000 7fa08b7fe640 10 monclient(hunting): my global_id is 4471 2024-03-29T10:43:28.488 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.504+0000 7fa08b7fe640 1 -- 172.21.15.84:0/558606490 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa08c149870 con 0x7fa08c0add40 2024-03-29T10:43:28.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.505+0000 7fa08b7fe640 1 -- 172.21.15.84:0/558606490 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1548218784 0 0) 0x7fa074003520 con 0x7fa08c0aad20 2024-03-29T10:43:28.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.505+0000 7fa08b7fe640 1 -- 172.21.15.84:0/558606490 >> v1:172.21.15.84:6790/0 conn(0x7fa08c0a9750 legacy=0x7fa08c149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:43:28.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.505+0000 7fa08b7fe640 1 -- 172.21.15.84:0/558606490 >> v1:172.21.15.86:6789/0 conn(0x7fa08c0add40 legacy=0x7fa08c143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:28.489 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.505+0000 7fa08b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:43:28.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.505+0000 7fa08b7fe640 1 monclient: found mon.noname-a 2024-03-29T10:43:28.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.505+0000 7fa08b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:28.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.505+0000 7fa08b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T10:43:28.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.505+0000 7fa08b7fe640 1 -- 172.21.15.84:0/558606490 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa08c14a6f0 con 0x7fa08c0aad20 2024-03-29T10:43:28.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.505+0000 7fa08b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:43:28.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.505+0000 7fa08b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:43:28.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.505+0000 7fa08b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:28.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.505+0000 7fa091d03640 5 monclient: authenticate success, global_id 4503 2024-03-29T10:43:28.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.505+0000 7fa091d03640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:28.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.505+0000 7fa08b7fe640 1 -- 172.21.15.84:0/558606490 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa07c001a10 con 0x7fa08c0aad20 2024-03-29T10:43:28.490 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.505+0000 7fa08b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:28.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.505+0000 7fa08b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:28.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.505+0000 7fa08b7fe640 10 monclient: dump: 2024-03-29T10:43:28.491 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:28.491 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:28.491 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:28.491 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:28.491 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:28.491 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:28.491 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:28.491 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:28.491 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:28.491 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:28.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.505+0000 7fa091d03640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:28.491 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.505+0000 7fa08b7fe640 1 -- 172.21.15.84:0/558606490 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa07c001c30 con 0x7fa08c0aad20 2024-03-29T10:43:28.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.505+0000 7fa08b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:28.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.505+0000 7fa091d03640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:28.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.505+0000 7fa091d03640 4 set_mon_vals no callback set 2024-03-29T10:43:28.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.506+0000 7fa091d03640 10 monclient: shutdown 2024-03-29T10:43:28.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.506+0000 7fa091d03640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:28.492 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.506+0000 7fa091d03640 1 -- 172.21.15.84:0/558606490 >> v1:172.21.15.84:6789/0 conn(0x7fa08c0aad20 legacy=0x7fa08c151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:28.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.507+0000 7fa091d03640 1 -- 172.21.15.84:0/558606490 shutdown_connections 2024-03-29T10:43:28.493 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.507+0000 7fa091d03640 1 -- 172.21.15.84:0/558606490 >> 172.21.15.84:0/558606490 conn(0x7fa08c0673e0 msgr2=0x7fa08c0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:28.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.508+0000 7fa091d03640 1 -- 172.21.15.84:0/558606490 shutdown_connections 2024-03-29T10:43:28.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.508+0000 7fa091d03640 1 -- 172.21.15.84:0/558606490 wait complete. 2024-03-29T10:43:28.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.509+0000 7fa091d03640 10 monclient: build_initial_monmap 2024-03-29T10:43:28.494 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.509+0000 7fa091d03640 10 monclient: monmap: 2024-03-29T10:43:28.494 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:43:28.494 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:28.494 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:43:28.494 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:43:28.494 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:28.494 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:28.494 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:43:28.494 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:43:28.494 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:43:28.494 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:28.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.509+0000 7fa091d03640 1 librados: starting msgr at 2024-03-29T10:43:28.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.509+0000 7fa091d03640 1 librados: starting objecter 2024-03-29T10:43:28.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.509+0000 7fa091d03640 1 Processor -- start 2024-03-29T10:43:28.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.510+0000 7fa091d03640 1 -- start start 2024-03-29T10:43:28.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.510+0000 7fa091d03640 1 librados: setting wanted keys 2024-03-29T10:43:28.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.510+0000 7fa091d03640 1 librados: calling monclient init 2024-03-29T10:43:28.495 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.510+0000 7fa091d03640 10 monclient: init 2024-03-29T10:43:28.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.511+0000 7fa091d03640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:28.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.511+0000 7fa091d03640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:43:28.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.511+0000 7fa091d03640 10 monclient(hunting): picked mon.noname-a con 0x7fa08c0a9750 addr v1:172.21.15.84:6789/0 2024-03-29T10:43:28.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.511+0000 7fa091d03640 10 monclient(hunting): picked mon.noname-c con 0x7fa08c0aad20 addr v1:172.21.15.84:6790/0 2024-03-29T10:43:28.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.511+0000 7fa091d03640 10 monclient(hunting): picked mon.noname-b con 0x7fa08c0add40 addr v1:172.21.15.86:6789/0 2024-03-29T10:43:28.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.511+0000 7fa091d03640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa08c1de5e0 con 0x7fa08c0a9750 2024-03-29T10:43:28.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.511+0000 7fa091d03640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa08c06fab0 con 0x7fa08c0add40 2024-03-29T10:43:28.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.511+0000 7fa091d03640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa08c144980 con 0x7fa08c0aad20 2024-03-29T10:43:28.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.511+0000 7fa091d03640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:28.496 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.511+0000 7fa091d03640 10 monclient(hunting): authenticate will time out at 1310.774250s 2024-03-29T10:43:28.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.514+0000 7fa091502640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa08c0add40 0x7fa08c1e68c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:49217/0 (socket says 172.21.15.84:49217) 2024-03-29T10:43:28.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.514+0000 7fa091502640 1 -- 172.21.15.84:0/72622366 learned_addr learned my addr 172.21.15.84:0/72622366 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:43:28.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.514+0000 7fa0897fa640 10 client.?.objecter ms_handle_connect 0x7fa08c0add40 2024-03-29T10:43:28.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.514+0000 7fa0897fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:28.498 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.514+0000 7fa0897fa640 1 -- 172.21.15.84:0/72622366 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807124636 0 0) 0x7fa08c06fab0 con 0x7fa08c0add40 2024-03-29T10:43:28.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.514+0000 7fa0897fa640 10 monclient(hunting): _negotiate 2024-03-29T10:43:28.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.514+0000 7fa0897fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:28.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.514+0000 7fa0897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:28.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.514+0000 7fa0897fa640 10 monclient(hunting): my global_id is 4474 2024-03-29T10:43:28.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.514+0000 7fa0897fa640 1 -- 172.21.15.84:0/72622366 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa078004eb0 con 0x7fa08c0add40 2024-03-29T10:43:28.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.515+0000 7fa0897fa640 10 client.?.objecter ms_handle_connect 0x7fa08c0a9750 2024-03-29T10:43:28.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.515+0000 7fa0897fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:28.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.515+0000 7fa0897fa640 1 -- 172.21.15.84:0/72622366 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3415308695 0 0) 0x7fa078004eb0 con 0x7fa08c0add40 2024-03-29T10:43:28.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.515+0000 7fa0897fa640 1 -- 172.21.15.84:0/72622366 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa08c06fab0 con 0x7fa08c0add40 2024-03-29T10:43:28.499 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.515+0000 7fa0897fa640 1 -- 172.21.15.84:0/72622366 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa07c004d40 con 0x7fa08c0add40 2024-03-29T10:43:28.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.515+0000 7fa0897fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:28.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.516+0000 7fa0897fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:28.501 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.516+0000 7fa0897fa640 10 monclient(hunting): dump: 2024-03-29T10:43:28.501 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:28.501 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:28.501 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:28.502 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:28.502 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:28.502 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:28.502 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:28.502 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:28.502 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:28.502 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:28.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.516+0000 7fa0897fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:28.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.516+0000 7fa0897fa640 1 -- 172.21.15.84:0/72622366 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4116730997 0 0) 0x7fa08c1de5e0 con 0x7fa08c0a9750 2024-03-29T10:43:28.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.516+0000 7fa0897fa640 10 monclient(hunting): _negotiate 2024-03-29T10:43:28.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.516+0000 7fa0897fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:28.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.516+0000 7fa0897fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:28.502 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.516+0000 7fa0897fa640 10 monclient(hunting): my global_id is 4506 2024-03-29T10:43:28.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.516+0000 7fa0897fa640 1 -- 172.21.15.84:0/72622366 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa078004eb0 con 0x7fa08c0a9750 2024-03-29T10:43:28.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.516+0000 7fa0897fa640 1 -- 172.21.15.84:0/72622366 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 489536794 0 0) 0x7fa08c06fab0 con 0x7fa08c0add40 2024-03-29T10:43:28.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.517+0000 7fa0897fa640 1 -- 172.21.15.84:0/72622366 >> v1:172.21.15.84:6790/0 conn(0x7fa08c0aad20 legacy=0x7fa08c13e170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:43:28.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.518+0000 7fa0897fa640 1 -- 172.21.15.84:0/72622366 >> v1:172.21.15.84:6789/0 conn(0x7fa08c0a9750 legacy=0x7fa08c1deac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:28.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.518+0000 7fa0897fa640 10 monclient: _finish_hunting 0 2024-03-29T10:43:28.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.518+0000 7fa0897fa640 1 monclient: found mon.b 2024-03-29T10:43:28.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.518+0000 7fa0897fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:28.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.518+0000 7fa0897fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:43:28.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.518+0000 7fa0897fa640 1 -- 172.21.15.84:0/72622366 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa08c144b10 con 0x7fa08c0add40 2024-03-29T10:43:28.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.518+0000 7fa0897fa640 10 monclient: _finish_auth 0 2024-03-29T10:43:28.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.518+0000 7fa0897fa640 10 monclient: _check_auth_tickets 2024-03-29T10:43:28.503 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.518+0000 7fa0897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:28.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.518+0000 7fa091d03640 5 monclient: authenticate success, global_id 4474 2024-03-29T10:43:28.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.518+0000 7fa091d03640 10 monclient: _renew_subs 2024-03-29T10:43:28.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.518+0000 7fa091d03640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:43:28.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.518+0000 7fa091d03640 1 -- 172.21.15.84:0/72622366 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa08c06ea30 con 0x7fa08c0add40 2024-03-29T10:43:28.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.518+0000 7fa091d03640 10 client.4474.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:28.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.518+0000 7fa091d03640 10 monclient: _renew_subs 2024-03-29T10:43:28.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.518+0000 7fa091d03640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:43:28.506 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.518+0000 7fa091d03640 1 -- 172.21.15.84:0/72622366 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa08c1df3b0 con 0x7fa08c0add40 2024-03-29T10:43:28.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.518+0000 7fa091d03640 1 librados: init done 2024-03-29T10:43:28.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.520+0000 7fa0897fa640 1 -- 172.21.15.84:0/72622366 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa07c005280 con 0x7fa08c0add40 2024-03-29T10:43:28.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.520+0000 7fa0897fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:28.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.520+0000 7fa0897fa640 1 -- 172.21.15.84:0/72622366 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa07c005470 con 0x7fa08c0add40 2024-03-29T10:43:28.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.520+0000 7fa0897fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:28.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.520+0000 7fa0897fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:43:28.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.520+0000 7fa08affd640 4 set_mon_vals no callback set 2024-03-29T10:43:28.507 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.521+0000 7fa0897fa640 10 monclient: dump: 2024-03-29T10:43:28.507 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:28.507 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:28.507 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:28.507 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:28.507 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:28.507 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:28.507 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:28.508 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:28.508 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:28.508 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:28.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.521+0000 7fa091d03640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:28.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.522+0000 7fa091d03640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:28.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.522+0000 7fa091d03640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:43:28.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.522+0000 7fa091d03640 1 -- 172.21.15.84:0/72622366 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa06c005080 con 0x7fa08c0add40 2024-03-29T10:43:28.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.522+0000 7fa0897fa640 1 -- 172.21.15.84:0/72622366 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7fa07c001da0 con 0x7fa08c0add40 2024-03-29T10:43:28.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.523+0000 7fa090d01640 10 monclient: get_auth_request con 0x7fa07806db40 auth_method 0 2024-03-29T10:43:28.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.523+0000 7fa0897fa640 1 -- 172.21.15.84:0/72622366 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7fa07c084e60 con 0x7fa08c0add40 2024-03-29T10:43:28.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.523+0000 7fa0897fa640 10 client.4474.objecter ms_dispatch 0x7fa08c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:28.508 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.523+0000 7fa0897fa640 3 client.4474.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:28.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.523+0000 7fa0897fa640 3 client.4474.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:28.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.523+0000 7fa0897fa640 20 client.4474.objecter dump_active .. 0 homeless 2024-03-29T10:43:28.509 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.523+0000 7fa0897fa640 10 client.4474.objecter ms_handle_connect 0x7fa07806db40 2024-03-29T10:43:28.511 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.528+0000 7fa0897fa640 1 -- 172.21.15.84:0/72622366 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa07c050740 con 0x7fa08c0add40 2024-03-29T10:43:28.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.528+0000 7fa0897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:28.512 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.528+0000 7fa0897fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:28.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.865+0000 7fa091d03640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:28.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.865+0000 7fa091d03640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:28.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.865+0000 7fa091d03640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:43:28.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.865+0000 7fa091d03640 1 -- 172.21.15.84:0/72622366 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa06c005330 con 0x7fa08c0add40 2024-03-29T10:43:28.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.867+0000 7fa0897fa640 1 -- 172.21.15.84:0/72622366 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7fa07c052670 con 0x7fa08c0add40 2024-03-29T10:43:28.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.867+0000 7fa0897fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:28.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.867+0000 7fa0897fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:28.850 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:43:28.851 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:28.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.870+0000 7fa091d03640 10 librados: watch_flush enter 2024-03-29T10:43:28.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.872+0000 7fa091d03640 10 librados: watch_flush exit 2024-03-29T10:43:28.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.872+0000 7fa091d03640 20 client.4474.objecter shutdown clearing up homeless session... 2024-03-29T10:43:28.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.872+0000 7fa091d03640 10 client.4474.objecter successfully canceled tick 2024-03-29T10:43:28.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.872+0000 7fa091d03640 1 -- 172.21.15.84:0/72622366 >> v1:172.21.15.86:6816/3337225337 conn(0x7fa07806db40 legacy=0x7fa078070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:28.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.872+0000 7fa091d03640 10 monclient: shutdown 2024-03-29T10:43:28.855 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.872+0000 7fa091d03640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:28.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.872+0000 7fa091d03640 1 -- 172.21.15.84:0/72622366 >> v1:172.21.15.86:6789/0 conn(0x7fa08c0add40 legacy=0x7fa08c1e68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:28.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.873+0000 7fa091d03640 1 -- 172.21.15.84:0/72622366 shutdown_connections 2024-03-29T10:43:28.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.873+0000 7fa091d03640 1 -- 172.21.15.84:0/72622366 >> 172.21.15.84:0/72622366 conn(0x7fa08c0673e0 msgr2=0x7fa08c13f6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:28.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.873+0000 7fa091d03640 1 -- 172.21.15.84:0/72622366 shutdown_connections 2024-03-29T10:43:28.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.873+0000 7fa091d03640 1 -- 172.21.15.84:0/72622366 wait complete. 2024-03-29T10:43:28.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:28.873+0000 7fa091d03640 1 librados: shutdown 2024-03-29T10:43:28.873 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:35.679 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:35.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.855+0000 7f99a68f6640 10 monclient: get_monmap_and_config 2024-03-29T10:43:35.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.855+0000 7f99a68f6640 10 monclient: build_initial_monmap 2024-03-29T10:43:35.839 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.855+0000 7f99a68f6640 10 monclient: monmap: 2024-03-29T10:43:35.839 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:43:35.839 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:35.839 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:43:35.856603+0000 2024-03-29T10:43:35.839 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:43:35.856603+0000 2024-03-29T10:43:35.839 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:35.839 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:35.839 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:43:35.839 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:43:35.839 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:43:35.839 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:35.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.856+0000 7f99a68f6640 1 Processor -- start 2024-03-29T10:43:35.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.857+0000 7f99a68f6640 1 -- start start 2024-03-29T10:43:35.840 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.857+0000 7f99a68f6640 10 monclient: init 2024-03-29T10:43:35.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.858+0000 7f99a68f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:35.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.858+0000 7f99a68f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:43:35.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.858+0000 7f99a68f6640 10 monclient(hunting): picked mon.noname-c con 0x7f99a01457f0 addr v1:172.21.15.86:6789/0 2024-03-29T10:43:35.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.858+0000 7f99a68f6640 10 monclient(hunting): picked mon.noname-b con 0x7f99a013f650 addr v1:172.21.15.84:6790/0 2024-03-29T10:43:35.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.858+0000 7f99a68f6640 10 monclient(hunting): picked mon.noname-a con 0x7f99a00a9d70 addr v1:172.21.15.84:6789/0 2024-03-29T10:43:35.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.858+0000 7f99a68f6640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a00aaef0 con 0x7f99a00a9d70 2024-03-29T10:43:35.842 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.858+0000 7f99a68f6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a0145bd0 con 0x7f99a01457f0 2024-03-29T10:43:35.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.858+0000 7f99a68f6640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a0145eb0 con 0x7f99a013f650 2024-03-29T10:43:35.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.858+0000 7f99a68f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:35.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.858+0000 7f99a68f6640 10 monclient(hunting): authenticate will time out at 1318.121338s 2024-03-29T10:43:35.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.858+0000 7f99a4e93640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f99a00a9d70 0x7f99a014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:55394/0 (socket says 172.21.15.84:55394) 2024-03-29T10:43:35.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.858+0000 7f99a4e93640 1 -- 172.21.15.84:0/1608832731 learned_addr learned my addr 172.21.15.84:0/1608832731 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:43:35.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.859+0000 7f999effd640 1 -- 172.21.15.84:0/1608832731 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 509599125 0 0) 0x7f99a0145eb0 con 0x7f99a013f650 2024-03-29T10:43:35.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.859+0000 7f999effd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:35.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.859+0000 7f999effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:35.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.859+0000 7f999effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:35.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.859+0000 7f999effd640 10 monclient(hunting): my global_id is 4601 2024-03-29T10:43:35.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.859+0000 7f999effd640 1 -- 172.21.15.84:0/1608832731 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9988003520 con 0x7f99a013f650 2024-03-29T10:43:35.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.859+0000 7f999effd640 1 -- 172.21.15.84:0/1608832731 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2064167260 0 0) 0x7f99a00aaef0 con 0x7f99a00a9d70 2024-03-29T10:43:35.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.859+0000 7f999effd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:35.843 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.859+0000 7f999effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:35.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.859+0000 7f999effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:35.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.859+0000 7f999effd640 10 monclient(hunting): my global_id is 4509 2024-03-29T10:43:35.844 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.859+0000 7f999effd640 1 -- 172.21.15.84:0/1608832731 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99a0145eb0 con 0x7f99a00a9d70 2024-03-29T10:43:35.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.859+0000 7f999effd640 1 -- 172.21.15.84:0/1608832731 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1401614562 0 0) 0x7f99a0145eb0 con 0x7f99a00a9d70 2024-03-29T10:43:35.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.859+0000 7f999effd640 1 -- 172.21.15.84:0/1608832731 >> v1:172.21.15.84:6790/0 conn(0x7f99a013f650 legacy=0x7f99a00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:35.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.859+0000 7f999effd640 1 -- 172.21.15.84:0/1608832731 >> v1:172.21.15.86:6789/0 conn(0x7f99a01457f0 legacy=0x7f99a013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:43:35.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.859+0000 7f999effd640 10 monclient: _finish_hunting 0 2024-03-29T10:43:35.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.859+0000 7f999effd640 1 monclient: found mon.noname-a 2024-03-29T10:43:35.845 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.859+0000 7f999effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:35.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.859+0000 7f999effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T10:43:35.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.859+0000 7f999effd640 1 -- 172.21.15.84:0/1608832731 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a0146340 con 0x7f99a00a9d70 2024-03-29T10:43:35.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.859+0000 7f999effd640 10 monclient: _finish_auth 0 2024-03-29T10:43:35.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.859+0000 7f999effd640 10 monclient: _check_auth_tickets 2024-03-29T10:43:35.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.859+0000 7f999effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:35.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.859+0000 7f999effd640 1 -- 172.21.15.84:0/1608832731 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f99900015f0 con 0x7f99a00a9d70 2024-03-29T10:43:35.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.859+0000 7f999effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:35.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.859+0000 7f999effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:35.846 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.859+0000 7f999effd640 10 monclient: dump: 2024-03-29T10:43:35.846 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:35.846 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:35.846 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:35.846 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:35.846 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:35.847 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:35.847 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:35.847 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:35.847 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:35.847 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:35.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.859+0000 7f99a68f6640 5 monclient: authenticate success, global_id 4509 2024-03-29T10:43:35.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.859+0000 7f99a68f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:35.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.860+0000 7f999effd640 1 -- 172.21.15.84:0/1608832731 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9990001750 con 0x7f99a00a9d70 2024-03-29T10:43:35.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.861+0000 7f999effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:35.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.861+0000 7f999effd640 1 -- 172.21.15.84:0/1608832731 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f9990001b60 con 0x7f99a00a9d70 2024-03-29T10:43:35.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.861+0000 7f999effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:35.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.861+0000 7f999effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:35.847 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.861+0000 7f999effd640 10 monclient: dump: 2024-03-29T10:43:35.847 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:35.847 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:35.848 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:35.848 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:35.848 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:35.848 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:35.848 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:35.848 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:35.848 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:35.848 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:35.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.861+0000 7f99a68f6640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:35.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.861+0000 7f99a68f6640 4 set_mon_vals no callback set 2024-03-29T10:43:35.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.862+0000 7f99a68f6640 10 monclient: shutdown 2024-03-29T10:43:35.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.862+0000 7f99a68f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:35.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.862+0000 7f99a68f6640 1 -- 172.21.15.84:0/1608832731 >> v1:172.21.15.84:6789/0 conn(0x7f99a00a9d70 legacy=0x7f99a014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:35.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.862+0000 7f99a68f6640 1 -- 172.21.15.84:0/1608832731 shutdown_connections 2024-03-29T10:43:35.848 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.862+0000 7f99a68f6640 1 -- 172.21.15.84:0/1608832731 >> 172.21.15.84:0/1608832731 conn(0x7f99a0066a00 msgr2=0x7f99a0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:35.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.862+0000 7f99a68f6640 1 -- 172.21.15.84:0/1608832731 shutdown_connections 2024-03-29T10:43:35.849 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.862+0000 7f99a68f6640 1 -- 172.21.15.84:0/1608832731 wait complete. 2024-03-29T10:43:35.850 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.864+0000 7f99a68f6640 10 monclient: build_initial_monmap 2024-03-29T10:43:35.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.864+0000 7f99a68f6640 10 monclient: monmap: 2024-03-29T10:43:35.851 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:43:35.851 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:35.851 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:43:35.851 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:43:35.851 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:35.851 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:35.851 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:43:35.851 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:43:35.851 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:43:35.851 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:35.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.864+0000 7f99a68f6640 1 librados: starting msgr at 2024-03-29T10:43:35.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.864+0000 7f99a68f6640 1 librados: starting objecter 2024-03-29T10:43:35.851 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.865+0000 7f99a68f6640 1 Processor -- start 2024-03-29T10:43:35.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.865+0000 7f99a68f6640 1 -- start start 2024-03-29T10:43:35.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.865+0000 7f99a68f6640 1 librados: setting wanted keys 2024-03-29T10:43:35.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.865+0000 7f99a68f6640 1 librados: calling monclient init 2024-03-29T10:43:35.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.865+0000 7f99a68f6640 10 monclient: init 2024-03-29T10:43:35.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.866+0000 7f99a68f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:35.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.866+0000 7f99a68f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:43:35.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.866+0000 7f99a68f6640 10 monclient(hunting): picked mon.noname-a con 0x7f99a00a9d70 addr v1:172.21.15.84:6789/0 2024-03-29T10:43:35.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.866+0000 7f99a68f6640 10 monclient(hunting): picked mon.noname-c con 0x7f99a013f650 addr v1:172.21.15.84:6790/0 2024-03-29T10:43:35.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.866+0000 7f99a68f6640 10 monclient(hunting): picked mon.noname-b con 0x7f99a01457f0 addr v1:172.21.15.86:6789/0 2024-03-29T10:43:35.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.866+0000 7f99a68f6640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a01424f0 con 0x7f99a00a9d70 2024-03-29T10:43:35.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.866+0000 7f99a68f6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a0142010 con 0x7f99a01457f0 2024-03-29T10:43:35.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.866+0000 7f99a68f6640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a01421a0 con 0x7f99a013f650 2024-03-29T10:43:35.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.866+0000 7f99a68f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:35.852 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.866+0000 7f99a68f6640 10 monclient(hunting): authenticate will time out at 1318.129799s 2024-03-29T10:43:35.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.866+0000 7f999ffff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f99a00a9d70 0x7f99a013b360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:56732/0 (socket says 172.21.15.84:56732) 2024-03-29T10:43:35.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.866+0000 7f999ffff640 1 -- 172.21.15.84:0/1045171082 learned_addr learned my addr 172.21.15.84:0/1045171082 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:43:35.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.866+0000 7f999cff9640 10 client.?.objecter ms_handle_connect 0x7f99a00a9d70 2024-03-29T10:43:35.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.866+0000 7f999cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:35.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.867+0000 7f999cff9640 10 client.?.objecter ms_handle_connect 0x7f99a01457f0 2024-03-29T10:43:35.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.867+0000 7f999cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:35.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.867+0000 7f999cff9640 10 client.?.objecter ms_handle_connect 0x7f99a013f650 2024-03-29T10:43:35.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.867+0000 7f999cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:35.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.867+0000 7f999cff9640 1 -- 172.21.15.84:0/1045171082 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 611051634 0 0) 0x7f99a01424f0 con 0x7f99a00a9d70 2024-03-29T10:43:35.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.867+0000 7f999cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:43:35.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.867+0000 7f999cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:35.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.867+0000 7f999cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:35.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.867+0000 7f999cff9640 10 monclient(hunting): my global_id is 4512 2024-03-29T10:43:35.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.867+0000 7f999cff9640 1 -- 172.21.15.84:0/1045171082 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f998c004eb0 con 0x7f99a00a9d70 2024-03-29T10:43:35.853 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.867+0000 7f999cff9640 1 -- 172.21.15.84:0/1045171082 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1747734407 0 0) 0x7f99a01421a0 con 0x7f99a013f650 2024-03-29T10:43:35.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.867+0000 7f999cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:43:35.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.867+0000 7f999cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:35.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.867+0000 7f999cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:35.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.867+0000 7f999cff9640 10 monclient(hunting): my global_id is 4604 2024-03-29T10:43:35.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.867+0000 7f999cff9640 1 -- 172.21.15.84:0/1045171082 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99a01424f0 con 0x7f99a013f650 2024-03-29T10:43:35.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.867+0000 7f999cff9640 1 -- 172.21.15.84:0/1045171082 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2080688732 0 0) 0x7f99a0142010 con 0x7f99a01457f0 2024-03-29T10:43:35.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.867+0000 7f999cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:43:35.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.867+0000 7f999cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:35.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.867+0000 7f999cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:35.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.867+0000 7f999cff9640 10 monclient(hunting): my global_id is 4477 2024-03-29T10:43:35.854 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.867+0000 7f999cff9640 1 -- 172.21.15.84:0/1045171082 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99a01421a0 con 0x7f99a01457f0 2024-03-29T10:43:35.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.868+0000 7f999cff9640 1 -- 172.21.15.84:0/1045171082 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1652821741 0 0) 0x7f99a01421a0 con 0x7f99a01457f0 2024-03-29T10:43:35.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.868+0000 7f999cff9640 1 -- 172.21.15.84:0/1045171082 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99a0142010 con 0x7f99a01457f0 2024-03-29T10:43:35.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.868+0000 7f999cff9640 1 -- 172.21.15.84:0/1045171082 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1971226846 0 0) 0x7f998c004eb0 con 0x7f99a00a9d70 2024-03-29T10:43:35.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.868+0000 7f999cff9640 1 -- 172.21.15.84:0/1045171082 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99a01421a0 con 0x7f99a00a9d70 2024-03-29T10:43:35.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.868+0000 7f999cff9640 1 -- 172.21.15.84:0/1045171082 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2920602836 0 0) 0x7f99a01424f0 con 0x7f99a013f650 2024-03-29T10:43:35.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.868+0000 7f999cff9640 1 -- 172.21.15.84:0/1045171082 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f998c004eb0 con 0x7f99a013f650 2024-03-29T10:43:35.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.868+0000 7f999cff9640 1 -- 172.21.15.84:0/1045171082 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f9990001a60 con 0x7f99a01457f0 2024-03-29T10:43:35.856 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.868+0000 7f999cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:35.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.868+0000 7f999cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:35.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.868+0000 7f999cff9640 10 monclient(hunting): dump: 2024-03-29T10:43:35.857 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:35.857 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:35.857 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:35.857 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:35.857 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:35.857 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:35.857 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:35.857 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:35.857 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:35.857 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:35.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.868+0000 7f999cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:35.857 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.868+0000 7f999cff9640 1 -- 172.21.15.84:0/1045171082 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f99980042e0 con 0x7f99a00a9d70 2024-03-29T10:43:35.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.868+0000 7f999cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:35.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.868+0000 7f999cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:35.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.868+0000 7f999cff9640 10 monclient(hunting): dump: 2024-03-29T10:43:35.858 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:35.858 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:35.858 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:35.858 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:35.858 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:35.858 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:35.858 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:35.858 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:35.858 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:35.858 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:35.858 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.868+0000 7f999cff9640 1 -- 172.21.15.84:0/1045171082 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f99940017c0 con 0x7f99a013f650 2024-03-29T10:43:35.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.868+0000 7f999cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:35.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.868+0000 7f999cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:35.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.868+0000 7f999cff9640 10 monclient(hunting): dump: 2024-03-29T10:43:35.859 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:35.859 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:35.859 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:35.859 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:35.859 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:35.859 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:35.859 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:35.859 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:35.859 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:35.859 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:35.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.869+0000 7f999cff9640 1 -- 172.21.15.84:0/1045171082 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2514616445 0 0) 0x7f99a01421a0 con 0x7f99a00a9d70 2024-03-29T10:43:35.859 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.869+0000 7f999cff9640 1 -- 172.21.15.84:0/1045171082 >> v1:172.21.15.84:6790/0 conn(0x7f99a013f650 legacy=0x7f99a01db750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:35.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.869+0000 7f999cff9640 1 -- 172.21.15.84:0/1045171082 >> v1:172.21.15.86:6789/0 conn(0x7f99a01457f0 legacy=0x7f99a01dbe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:35.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.869+0000 7f999cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:43:35.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.869+0000 7f999cff9640 1 monclient: found mon.a 2024-03-29T10:43:35.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.869+0000 7f999cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:35.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.869+0000 7f999cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:43:35.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.869+0000 7f999cff9640 1 -- 172.21.15.84:0/1045171082 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a006e100 con 0x7f99a00a9d70 2024-03-29T10:43:35.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.869+0000 7f999cff9640 10 monclient: _finish_auth 0 2024-03-29T10:43:35.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.869+0000 7f999cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:43:35.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.869+0000 7f999cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:35.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.869+0000 7f99a68f6640 5 monclient: authenticate success, global_id 4512 2024-03-29T10:43:35.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.869+0000 7f99a68f6640 10 monclient: _renew_subs 2024-03-29T10:43:35.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.869+0000 7f99a68f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:43:35.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.869+0000 7f99a68f6640 1 -- 172.21.15.84:0/1045171082 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99a006e350 con 0x7f99a00a9d70 2024-03-29T10:43:35.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.869+0000 7f99a68f6640 10 client.4512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:35.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.869+0000 7f99a68f6640 10 monclient: _renew_subs 2024-03-29T10:43:35.860 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.869+0000 7f99a68f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:43:35.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.869+0000 7f99a68f6640 1 -- 172.21.15.84:0/1045171082 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f99a01e7360 con 0x7f99a00a9d70 2024-03-29T10:43:35.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.869+0000 7f99a68f6640 1 librados: init done 2024-03-29T10:43:35.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.869+0000 7f999cff9640 1 -- 172.21.15.84:0/1045171082 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9998004440 con 0x7f99a00a9d70 2024-03-29T10:43:35.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.869+0000 7f999cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:35.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.869+0000 7f999cff9640 1 -- 172.21.15.84:0/1045171082 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f99980013f0 con 0x7f99a00a9d70 2024-03-29T10:43:35.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.869+0000 7f999cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:35.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.869+0000 7f999cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:35.861 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.869+0000 7f999cff9640 10 monclient: dump: 2024-03-29T10:43:35.861 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:35.861 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:35.861 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:35.861 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:35.861 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:35.861 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:35.861 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:35.861 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:35.862 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:35.862 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:35.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.869+0000 7f999dffb640 4 set_mon_vals no callback set 2024-03-29T10:43:35.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.871+0000 7f99a68f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:35.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.871+0000 7f99a68f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:35.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.871+0000 7f99a68f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:43:35.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.871+0000 7f99a68f6640 1 -- 172.21.15.84:0/1045171082 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9980005080 con 0x7f99a00a9d70 2024-03-29T10:43:35.862 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.873+0000 7f999cff9640 1 -- 172.21.15.84:0/1045171082 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f9998001550 con 0x7f99a00a9d70 2024-03-29T10:43:35.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.874+0000 7f999cff9640 1 -- 172.21.15.84:0/1045171082 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f9998084c90 con 0x7f99a00a9d70 2024-03-29T10:43:35.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.874+0000 7f999cff9640 10 client.4512.objecter ms_dispatch 0x7f99a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:35.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.874+0000 7f999cff9640 3 client.4512.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:35.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.874+0000 7f999cff9640 3 client.4512.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:35.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.874+0000 7f999cff9640 20 client.4512.objecter dump_active .. 0 homeless 2024-03-29T10:43:35.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.875+0000 7f999f7fe640 10 monclient: get_auth_request con 0x7f998c06e240 auth_method 0 2024-03-29T10:43:35.863 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.875+0000 7f999cff9640 10 client.4512.objecter ms_handle_connect 0x7f998c06e240 2024-03-29T10:43:35.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.881+0000 7f999cff9640 1 -- 172.21.15.84:0/1045171082 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f99980505d0 con 0x7f99a00a9d70 2024-03-29T10:43:35.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.881+0000 7f999cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:35.864 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:35.881+0000 7f999cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:36.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:36.237+0000 7f99a68f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:36.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:36.237+0000 7f99a68f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:36.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:36.237+0000 7f99a68f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:43:36.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:36.237+0000 7f99a68f6640 1 -- 172.21.15.84:0/1045171082 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9980005330 con 0x7f99a00a9d70 2024-03-29T10:43:36.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:36.238+0000 7f999cff9640 1 -- 172.21.15.84:0/1045171082 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f9998052500 con 0x7f99a00a9d70 2024-03-29T10:43:36.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:36.238+0000 7f999cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:36.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:36.238+0000 7f999cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:36.222 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:43:36.222 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:36.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:36.242+0000 7f99a68f6640 10 librados: watch_flush enter 2024-03-29T10:43:36.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:36.242+0000 7f99a68f6640 10 librados: watch_flush exit 2024-03-29T10:43:36.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:36.242+0000 7f99a68f6640 20 client.4512.objecter shutdown clearing up homeless session... 2024-03-29T10:43:36.226 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:36.242+0000 7f99a68f6640 10 client.4512.objecter successfully canceled tick 2024-03-29T10:43:36.226 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:36.242+0000 7f99a68f6640 1 -- 172.21.15.84:0/1045171082 >> v1:172.21.15.86:6816/3337225337 conn(0x7f998c06e240 legacy=0x7f998c070700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:36.226 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:36.242+0000 7f99a68f6640 10 monclient: shutdown 2024-03-29T10:43:36.226 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:36.242+0000 7f99a68f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:36.226 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:36.242+0000 7f99a68f6640 1 -- 172.21.15.84:0/1045171082 >> v1:172.21.15.84:6789/0 conn(0x7f99a00a9d70 legacy=0x7f99a013b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:36.226 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:36.242+0000 7f99a68f6640 1 -- 172.21.15.84:0/1045171082 shutdown_connections 2024-03-29T10:43:36.226 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:36.242+0000 7f99a68f6640 1 -- 172.21.15.84:0/1045171082 >> 172.21.15.84:0/1045171082 conn(0x7f99a0066a00 msgr2=0x7f99a00b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:36.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:36.243+0000 7f99a68f6640 1 -- 172.21.15.84:0/1045171082 shutdown_connections 2024-03-29T10:43:36.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:36.243+0000 7f99a68f6640 1 -- 172.21.15.84:0/1045171082 wait complete. 2024-03-29T10:43:36.227 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:36.244+0000 7f99a68f6640 1 librados: shutdown 2024-03-29T10:43:36.242 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:43.049 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:43.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.215+0000 7ff9e0d2e640 10 monclient: get_monmap_and_config 2024-03-29T10:43:43.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.215+0000 7ff9e0d2e640 10 monclient: build_initial_monmap 2024-03-29T10:43:43.199 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.215+0000 7ff9e0d2e640 10 monclient: monmap: 2024-03-29T10:43:43.199 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:43:43.199 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:43.199 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:43:43.216696+0000 2024-03-29T10:43:43.199 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:43:43.216696+0000 2024-03-29T10:43:43.199 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:43.199 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:43.199 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:43:43.199 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:43:43.200 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:43:43.200 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:43.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.217+0000 7ff9e0d2e640 1 Processor -- start 2024-03-29T10:43:43.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.217+0000 7ff9e0d2e640 1 -- start start 2024-03-29T10:43:43.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.217+0000 7ff9e0d2e640 10 monclient: init 2024-03-29T10:43:43.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.218+0000 7ff9e0d2e640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:43.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.218+0000 7ff9e0d2e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:43:43.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.218+0000 7ff9e0d2e640 10 monclient(hunting): picked mon.noname-c con 0x7ff9dc0b46b0 addr v1:172.21.15.86:6789/0 2024-03-29T10:43:43.201 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.218+0000 7ff9e0d2e640 10 monclient(hunting): picked mon.noname-a con 0x7ff9dc0aeae0 addr v1:172.21.15.84:6789/0 2024-03-29T10:43:43.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.218+0000 7ff9d3fff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7ff9dc0aeae0 0x7ff9dc13f300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:51704/0 (socket says 172.21.15.84:51704) 2024-03-29T10:43:43.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.218+0000 7ff9d3fff640 1 -- 172.21.15.84:0/932255744 learned_addr learned my addr 172.21.15.84:0/932255744 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:43:43.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.218+0000 7ff9e0d2e640 10 monclient(hunting): picked mon.noname-b con 0x7ff9dc13fa10 addr v1:172.21.15.84:6790/0 2024-03-29T10:43:43.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.218+0000 7ff9e0d2e640 1 -- 172.21.15.84:0/932255744 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9dc0b06a0 con 0x7ff9dc0aeae0 2024-03-29T10:43:43.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.218+0000 7ff9e0d2e640 1 -- 172.21.15.84:0/932255744 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9dc0b4230 con 0x7ff9dc0b46b0 2024-03-29T10:43:43.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.218+0000 7ff9e0d2e640 1 -- 172.21.15.84:0/932255744 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9dc0b0910 con 0x7ff9dc13fa10 2024-03-29T10:43:43.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.218+0000 7ff9e0d2e640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:43.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.218+0000 7ff9e0d2e640 10 monclient(hunting): authenticate will time out at 1325.481658s 2024-03-29T10:43:43.204 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 1 -- 172.21.15.84:0/932255744 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1531650924 0 0) 0x7ff9dc0b06a0 con 0x7ff9dc0aeae0 2024-03-29T10:43:43.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:43:43.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:43.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:43.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 10 monclient(hunting): my global_id is 4515 2024-03-29T10:43:43.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 1 -- 172.21.15.84:0/932255744 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9c8003520 con 0x7ff9dc0aeae0 2024-03-29T10:43:43.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 1 -- 172.21.15.84:0/932255744 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3559353703 0 0) 0x7ff9dc0b0910 con 0x7ff9dc13fa10 2024-03-29T10:43:43.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:43:43.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:43.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:43.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 10 monclient(hunting): my global_id is 4607 2024-03-29T10:43:43.205 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 1 -- 172.21.15.84:0/932255744 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9dc0b06a0 con 0x7ff9dc13fa10 2024-03-29T10:43:43.206 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 1 -- 172.21.15.84:0/932255744 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1655091349 0 0) 0x7ff9dc0b4230 con 0x7ff9dc0b46b0 2024-03-29T10:43:43.206 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:43:43.206 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:43.206 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:43.206 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 10 monclient(hunting): my global_id is 4480 2024-03-29T10:43:43.206 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 1 -- 172.21.15.84:0/932255744 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9dc0b0910 con 0x7ff9dc0b46b0 2024-03-29T10:43:43.206 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 1 -- 172.21.15.84:0/932255744 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 637015132 0 0) 0x7ff9dc0b06a0 con 0x7ff9dc13fa10 2024-03-29T10:43:43.206 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 1 -- 172.21.15.84:0/932255744 >> v1:172.21.15.86:6789/0 conn(0x7ff9dc0b46b0 legacy=0x7ff9dc0b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:43.206 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 1 -- 172.21.15.84:0/932255744 >> v1:172.21.15.84:6789/0 conn(0x7ff9dc0aeae0 legacy=0x7ff9dc13f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:43.206 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:43:43.206 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 1 monclient: found mon.noname-b 2024-03-29T10:43:43.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:43.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:43:43.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 1 -- 172.21.15.84:0/932255744 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9dc0af6d0 con 0x7ff9dc13fa10 2024-03-29T10:43:43.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 10 monclient: _finish_auth 0 2024-03-29T10:43:43.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:43:43.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:43.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 1 -- 172.21.15.84:0/932255744 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7ff9d40019f0 con 0x7ff9dc13fa10 2024-03-29T10:43:43.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:43.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:43.207 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9d9d9b640 10 monclient: dump: 2024-03-29T10:43:43.207 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:43.207 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:43.208 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:43.208 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:43.208 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:43.208 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:43.208 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:43.208 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:43.208 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:43.208 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:43.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9e0d2e640 5 monclient: authenticate success, global_id 4607 2024-03-29T10:43:43.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.219+0000 7ff9e0d2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:43:43.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.220+0000 7ff9d9d9b640 1 -- 172.21.15.84:0/932255744 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9d4001b50 con 0x7ff9dc13fa10 2024-03-29T10:43:43.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.220+0000 7ff9d9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:43.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.220+0000 7ff9d9d9b640 1 -- 172.21.15.84:0/932255744 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7ff9d4001f60 con 0x7ff9dc13fa10 2024-03-29T10:43:43.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.220+0000 7ff9d9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:43.208 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.220+0000 7ff9d9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:43.209 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.220+0000 7ff9d9d9b640 10 monclient: dump: 2024-03-29T10:43:43.209 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:43.209 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:43.209 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:43.209 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:43.209 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:43.209 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:43.209 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:43.209 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:43.209 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:43.209 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:43.209 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.220+0000 7ff9e0d2e640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:43.209 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.220+0000 7ff9e0d2e640 4 set_mon_vals no callback set 2024-03-29T10:43:43.209 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.220+0000 7ff9e0d2e640 10 monclient: shutdown 2024-03-29T10:43:43.209 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.220+0000 7ff9e0d2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:43.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.220+0000 7ff9e0d2e640 1 -- 172.21.15.84:0/932255744 >> v1:172.21.15.84:6790/0 conn(0x7ff9dc13fa10 legacy=0x7ff9dc14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:43.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.221+0000 7ff9e0d2e640 1 -- 172.21.15.84:0/932255744 shutdown_connections 2024-03-29T10:43:43.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.221+0000 7ff9e0d2e640 1 -- 172.21.15.84:0/932255744 >> 172.21.15.84:0/932255744 conn(0x7ff9dc066a00 msgr2=0x7ff9dc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:43.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.221+0000 7ff9e0d2e640 1 -- 172.21.15.84:0/932255744 shutdown_connections 2024-03-29T10:43:43.210 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.221+0000 7ff9e0d2e640 1 -- 172.21.15.84:0/932255744 wait complete. 2024-03-29T10:43:43.214 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.222+0000 7ff9e0d2e640 10 monclient: build_initial_monmap 2024-03-29T10:43:43.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.222+0000 7ff9e0d2e640 10 monclient: monmap: 2024-03-29T10:43:43.215 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:43:43.215 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:43.215 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:43:43.215 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:43:43.215 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:43.215 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:43.215 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:43:43.215 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:43:43.215 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:43:43.215 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:43.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.222+0000 7ff9e0d2e640 1 librados: starting msgr at 2024-03-29T10:43:43.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.222+0000 7ff9e0d2e640 1 librados: starting objecter 2024-03-29T10:43:43.215 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.222+0000 7ff9e0d2e640 1 Processor -- start 2024-03-29T10:43:43.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.223+0000 7ff9e0d2e640 1 -- start start 2024-03-29T10:43:43.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.223+0000 7ff9e0d2e640 1 librados: setting wanted keys 2024-03-29T10:43:43.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.223+0000 7ff9e0d2e640 1 librados: calling monclient init 2024-03-29T10:43:43.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.223+0000 7ff9e0d2e640 10 monclient: init 2024-03-29T10:43:43.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.224+0000 7ff9e0d2e640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:43.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.224+0000 7ff9e0d2e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:43:43.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.224+0000 7ff9e0d2e640 10 monclient(hunting): picked mon.noname-b con 0x7ff9dc0aeae0 addr v1:172.21.15.86:6789/0 2024-03-29T10:43:43.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.224+0000 7ff9e0d2e640 10 monclient(hunting): picked mon.noname-a con 0x7ff9dc0b46b0 addr v1:172.21.15.84:6789/0 2024-03-29T10:43:43.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.224+0000 7ff9e0d2e640 10 monclient(hunting): picked mon.noname-c con 0x7ff9dc13fa10 addr v1:172.21.15.84:6790/0 2024-03-29T10:43:43.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.224+0000 7ff9e0d2e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9dc1d9710 con 0x7ff9dc0b46b0 2024-03-29T10:43:43.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.224+0000 7ff9e0d2e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9dc1459c0 con 0x7ff9dc0aeae0 2024-03-29T10:43:43.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.224+0000 7ff9e0d2e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9dc1448b0 con 0x7ff9dc13fa10 2024-03-29T10:43:43.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.224+0000 7ff9e0d2e640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:43.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.224+0000 7ff9e0d2e640 10 monclient(hunting): authenticate will time out at 1325.487363s 2024-03-29T10:43:43.216 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.224+0000 7ff9dad9d640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7ff9dc13fa10 0x7ff9dc1dc1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:56152/0 (socket says 172.21.15.84:56152) 2024-03-29T10:43:43.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.224+0000 7ff9dad9d640 1 -- 172.21.15.84:0/3224729900 learned_addr learned my addr 172.21.15.84:0/3224729900 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:43:43.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.224+0000 7ff9d2ffd640 10 client.?.objecter ms_handle_connect 0x7ff9dc13fa10 2024-03-29T10:43:43.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.224+0000 7ff9d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:43.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.224+0000 7ff9d2ffd640 1 -- 172.21.15.84:0/3224729900 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2331709144 0 0) 0x7ff9dc1448b0 con 0x7ff9dc13fa10 2024-03-29T10:43:43.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.224+0000 7ff9d2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:43.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.224+0000 7ff9d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:43.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.224+0000 7ff9d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:43.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.224+0000 7ff9d2ffd640 10 monclient(hunting): my global_id is 4610 2024-03-29T10:43:43.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.224+0000 7ff9d2ffd640 1 -- 172.21.15.84:0/3224729900 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9bc004eb0 con 0x7ff9dc13fa10 2024-03-29T10:43:43.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.224+0000 7ff9d2ffd640 10 client.?.objecter ms_handle_connect 0x7ff9dc0b46b0 2024-03-29T10:43:43.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.224+0000 7ff9d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:43.217 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 10 client.?.objecter ms_handle_connect 0x7ff9dc0aeae0 2024-03-29T10:43:43.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:43.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 1 -- 172.21.15.84:0/3224729900 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2239844134 0 0) 0x7ff9bc004eb0 con 0x7ff9dc13fa10 2024-03-29T10:43:43.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 1 -- 172.21.15.84:0/3224729900 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9bc005300 con 0x7ff9dc13fa10 2024-03-29T10:43:43.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 1 -- 172.21.15.84:0/3224729900 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7ff9d4001f60 con 0x7ff9dc13fa10 2024-03-29T10:43:43.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:43.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:43.218 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 10 monclient(hunting): dump: 2024-03-29T10:43:43.218 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:43.218 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:43.219 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:43.219 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:43.219 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:43.219 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:43.219 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:43.219 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:43.219 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:43.219 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:43.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:43.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 1 -- 172.21.15.84:0/3224729900 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 624627746 0 0) 0x7ff9dc1459c0 con 0x7ff9dc0aeae0 2024-03-29T10:43:43.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:43.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:43.219 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:43.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 10 monclient(hunting): my global_id is 4483 2024-03-29T10:43:43.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 1 -- 172.21.15.84:0/3224729900 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9bc005850 con 0x7ff9dc0aeae0 2024-03-29T10:43:43.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 1 -- 172.21.15.84:0/3224729900 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1368069913 0 0) 0x7ff9dc1d9710 con 0x7ff9dc0b46b0 2024-03-29T10:43:43.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:43:43.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:43.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:43.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 10 monclient(hunting): my global_id is 4518 2024-03-29T10:43:43.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 1 -- 172.21.15.84:0/3224729900 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9dc1459c0 con 0x7ff9dc0b46b0 2024-03-29T10:43:43.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 1 -- 172.21.15.84:0/3224729900 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1152248767 0 0) 0x7ff9bc005300 con 0x7ff9dc13fa10 2024-03-29T10:43:43.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 1 -- 172.21.15.84:0/3224729900 >> v1:172.21.15.86:6789/0 conn(0x7ff9dc0aeae0 legacy=0x7ff9dc145090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:43.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 1 -- 172.21.15.84:0/3224729900 >> v1:172.21.15.84:6789/0 conn(0x7ff9dc0b46b0 legacy=0x7ff9dc1dbac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:43.220 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:43:43.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 1 monclient: found mon.c 2024-03-29T10:43:43.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:43.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:43:43.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 1 -- 172.21.15.84:0/3224729900 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9dc13b920 con 0x7ff9dc13fa10 2024-03-29T10:43:43.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:43:43.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:43:43.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.225+0000 7ff9d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:43.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.226+0000 7ff9e0d2e640 5 monclient: authenticate success, global_id 4610 2024-03-29T10:43:43.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.226+0000 7ff9d2ffd640 1 -- 172.21.15.84:0/3224729900 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9d4003c20 con 0x7ff9dc13fa10 2024-03-29T10:43:43.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.226+0000 7ff9d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:43.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.226+0000 7ff9d2ffd640 1 -- 172.21.15.84:0/3224729900 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7ff9d4001f60 con 0x7ff9dc13fa10 2024-03-29T10:43:43.221 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.226+0000 7ff9d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:43.222 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.226+0000 7ff9d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:43.222 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.226+0000 7ff9d2ffd640 10 monclient: dump: 2024-03-29T10:43:43.222 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:43.222 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:43.222 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:43.222 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:43.222 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:43.222 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:43.222 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:43.222 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:43.222 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:43.223 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:43.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.226+0000 7ff9d959a640 4 set_mon_vals no callback set 2024-03-29T10:43:43.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.226+0000 7ff9e0d2e640 10 monclient: _renew_subs 2024-03-29T10:43:43.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.226+0000 7ff9e0d2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:43:43.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.226+0000 7ff9e0d2e640 1 -- 172.21.15.84:0/3224729900 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9dc13bb70 con 0x7ff9dc13fa10 2024-03-29T10:43:43.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.226+0000 7ff9e0d2e640 10 client.4610.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:43.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.226+0000 7ff9e0d2e640 10 monclient: _renew_subs 2024-03-29T10:43:43.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.226+0000 7ff9e0d2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:43:43.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.226+0000 7ff9e0d2e640 1 -- 172.21.15.84:0/3224729900 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9dc1e76f0 con 0x7ff9dc13fa10 2024-03-29T10:43:43.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.226+0000 7ff9e0d2e640 1 librados: init done 2024-03-29T10:43:43.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.227+0000 7ff9e0d2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:43.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.228+0000 7ff9e0d2e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:43.223 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.228+0000 7ff9e0d2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:43:43.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.228+0000 7ff9e0d2e640 1 -- 172.21.15.84:0/3224729900 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9dc075f10 con 0x7ff9dc13fa10 2024-03-29T10:43:43.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.229+0000 7ff9d2ffd640 1 -- 172.21.15.84:0/3224729900 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7ff9d4003d90 con 0x7ff9dc13fa10 2024-03-29T10:43:43.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.230+0000 7ff9d2ffd640 1 -- 172.21.15.84:0/3224729900 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7ff9d40548e0 con 0x7ff9dc13fa10 2024-03-29T10:43:43.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.230+0000 7ff9d2ffd640 10 client.4610.objecter ms_dispatch 0x7ff9dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:43.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.230+0000 7ff9d2ffd640 3 client.4610.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:43.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.230+0000 7ff9d2ffd640 3 client.4610.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:43.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.230+0000 7ff9d2ffd640 20 client.4610.objecter dump_active .. 0 homeless 2024-03-29T10:43:43.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.230+0000 7ff9da59c640 10 monclient: get_auth_request con 0x7ff9bc06e440 auth_method 0 2024-03-29T10:43:43.224 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.230+0000 7ff9d2ffd640 10 client.4610.objecter ms_handle_connect 0x7ff9bc06e440 2024-03-29T10:43:43.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.238+0000 7ff9d2ffd640 1 -- 172.21.15.84:0/3224729900 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff9d4050b80 con 0x7ff9dc13fa10 2024-03-29T10:43:43.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.238+0000 7ff9d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:43.225 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.238+0000 7ff9d2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:43.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.594+0000 7ff9e0d2e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:43.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.594+0000 7ff9e0d2e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:43.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.594+0000 7ff9e0d2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:43:43.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.594+0000 7ff9e0d2e640 1 -- 172.21.15.84:0/3224729900 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9dc13bd00 con 0x7ff9dc13fa10 2024-03-29T10:43:43.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.595+0000 7ff9d2ffd640 1 -- 172.21.15.84:0/3224729900 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7ff9d4050b80 con 0x7ff9dc13fa10 2024-03-29T10:43:43.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.595+0000 7ff9d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:43.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.595+0000 7ff9d2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:43.578 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:43:43.578 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:43.582 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.598+0000 7ff9e0d2e640 10 librados: watch_flush enter 2024-03-29T10:43:43.582 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.598+0000 7ff9e0d2e640 10 librados: watch_flush exit 2024-03-29T10:43:43.582 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.598+0000 7ff9e0d2e640 20 client.4610.objecter shutdown clearing up homeless session... 2024-03-29T10:43:43.582 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.598+0000 7ff9e0d2e640 10 client.4610.objecter successfully canceled tick 2024-03-29T10:43:43.582 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.598+0000 7ff9e0d2e640 1 -- 172.21.15.84:0/3224729900 >> v1:172.21.15.86:6816/3337225337 conn(0x7ff9bc06e440 legacy=0x7ff9bc070900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:43.582 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.598+0000 7ff9e0d2e640 10 monclient: shutdown 2024-03-29T10:43:43.582 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.598+0000 7ff9e0d2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:43.582 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.598+0000 7ff9e0d2e640 1 -- 172.21.15.84:0/3224729900 >> v1:172.21.15.84:6790/0 conn(0x7ff9dc13fa10 legacy=0x7ff9dc1dc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:43.582 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.599+0000 7ff9e0d2e640 1 -- 172.21.15.84:0/3224729900 shutdown_connections 2024-03-29T10:43:43.582 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.599+0000 7ff9e0d2e640 1 -- 172.21.15.84:0/3224729900 >> 172.21.15.84:0/3224729900 conn(0x7ff9dc066a00 msgr2=0x7ff9dc13f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:43.583 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.599+0000 7ff9e0d2e640 1 -- 172.21.15.84:0/3224729900 shutdown_connections 2024-03-29T10:43:43.583 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.599+0000 7ff9e0d2e640 1 -- 172.21.15.84:0/3224729900 wait complete. 2024-03-29T10:43:43.583 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:43.599+0000 7ff9e0d2e640 1 librados: shutdown 2024-03-29T10:43:43.596 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:50.403 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:50.550 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.566+0000 7ffa6aa5f640 10 monclient: get_monmap_and_config 2024-03-29T10:43:50.550 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.566+0000 7ffa6aa5f640 10 monclient: build_initial_monmap 2024-03-29T10:43:50.550 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.567+0000 7ffa6aa5f640 10 monclient: monmap: 2024-03-29T10:43:50.550 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:43:50.550 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:50.550 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:43:50.567944+0000 2024-03-29T10:43:50.550 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:43:50.567944+0000 2024-03-29T10:43:50.550 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:50.550 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:50.551 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:43:50.551 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:43:50.551 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:43:50.551 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:50.552 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.568+0000 7ffa6aa5f640 1 Processor -- start 2024-03-29T10:43:50.553 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.569+0000 7ffa6aa5f640 1 -- start start 2024-03-29T10:43:50.553 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.569+0000 7ffa6aa5f640 10 monclient: init 2024-03-29T10:43:50.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.570+0000 7ffa6aa5f640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:50.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.570+0000 7ffa6aa5f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:43:50.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.570+0000 7ffa6aa5f640 10 monclient(hunting): picked mon.noname-b con 0x7ffa5c006a70 addr v1:172.21.15.84:6790/0 2024-03-29T10:43:50.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.570+0000 7ffa6aa5f640 10 monclient(hunting): picked mon.noname-c con 0x7ffa5c02b100 addr v1:172.21.15.86:6789/0 2024-03-29T10:43:50.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.570+0000 7ffa6aa5f640 10 monclient(hunting): picked mon.noname-a con 0x7ffa5c029c30 addr v1:172.21.15.84:6789/0 2024-03-29T10:43:50.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.570+0000 7ffa6aa5f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa5c02a270 con 0x7ffa5c029c30 2024-03-29T10:43:50.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.570+0000 7ffa6aa5f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa5c02c440 con 0x7ffa5c02b100 2024-03-29T10:43:50.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.570+0000 7ffa6aa5f640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa5c0c9440 con 0x7ffa5c006a70 2024-03-29T10:43:50.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.570+0000 7ffa6aa5f640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:50.554 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.570+0000 7ffa6aa5f640 10 monclient(hunting): authenticate will time out at 1332.833330s 2024-03-29T10:43:50.555 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.571+0000 7ffa69a5d640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7ffa5c006a70 0x7ffa5c0c8d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:55378/0 (socket says 172.21.15.84:55378) 2024-03-29T10:43:50.555 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.571+0000 7ffa69a5d640 1 -- 172.21.15.84:0/3653466636 learned_addr learned my addr 172.21.15.84:0/3653466636 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:43:50.561 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.575+0000 7ffa68a5b640 1 -- 172.21.15.84:0/3653466636 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1583866826 0 0) 0x7ffa5c0c9440 con 0x7ffa5c006a70 2024-03-29T10:43:50.561 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.575+0000 7ffa68a5b640 10 monclient(hunting): _negotiate 2024-03-29T10:43:50.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.575+0000 7ffa68a5b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:50.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.575+0000 7ffa68a5b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:50.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.575+0000 7ffa68a5b640 10 monclient(hunting): my global_id is 4613 2024-03-29T10:43:50.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.575+0000 7ffa68a5b640 1 -- 172.21.15.84:0/3653466636 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa50003520 con 0x7ffa5c006a70 2024-03-29T10:43:50.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.575+0000 7ffa68a5b640 1 -- 172.21.15.84:0/3653466636 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3751189749 0 0) 0x7ffa50003520 con 0x7ffa5c006a70 2024-03-29T10:43:50.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.575+0000 7ffa68a5b640 1 -- 172.21.15.84:0/3653466636 >> v1:172.21.15.86:6789/0 conn(0x7ffa5c02b100 legacy=0x7ffa5c0cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:43:50.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.575+0000 7ffa68a5b640 1 -- 172.21.15.84:0/3653466636 >> v1:172.21.15.84:6789/0 conn(0x7ffa5c029c30 legacy=0x7ffa5c0d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:43:50.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.575+0000 7ffa68a5b640 10 monclient: _finish_hunting 0 2024-03-29T10:43:50.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.575+0000 7ffa68a5b640 1 monclient: found mon.noname-b 2024-03-29T10:43:50.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.575+0000 7ffa68a5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:50.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.575+0000 7ffa68a5b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:43:50.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.575+0000 7ffa68a5b640 1 -- 172.21.15.84:0/3653466636 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa5c0c95d0 con 0x7ffa5c006a70 2024-03-29T10:43:50.562 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.575+0000 7ffa68a5b640 10 monclient: _finish_auth 0 2024-03-29T10:43:50.563 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.575+0000 7ffa68a5b640 10 monclient: _check_auth_tickets 2024-03-29T10:43:50.563 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.575+0000 7ffa68a5b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:50.563 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.575+0000 7ffa68a5b640 1 -- 172.21.15.84:0/3653466636 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7ffa60001690 con 0x7ffa5c006a70 2024-03-29T10:43:50.563 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.575+0000 7ffa68a5b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:50.563 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.575+0000 7ffa68a5b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:43:50.563 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.575+0000 7ffa68a5b640 10 monclient: dump: 2024-03-29T10:43:50.563 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:50.563 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:50.563 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:50.563 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:50.563 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:50.563 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:50.563 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:50.563 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:50.563 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:50.564 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:50.564 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.576+0000 7ffa68a5b640 1 -- 172.21.15.84:0/3653466636 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa60001860 con 0x7ffa5c006a70 2024-03-29T10:43:50.564 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.576+0000 7ffa68a5b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:50.564 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.576+0000 7ffa68a5b640 1 -- 172.21.15.84:0/3653466636 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7ffa60001cb0 con 0x7ffa5c006a70 2024-03-29T10:43:50.564 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.576+0000 7ffa68a5b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:50.564 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.576+0000 7ffa68a5b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:50.564 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.576+0000 7ffa68a5b640 10 monclient: dump: 2024-03-29T10:43:50.564 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:50.564 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:50.564 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:50.564 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:50.564 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:50.564 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:50.564 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:50.564 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:50.564 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:50.565 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:50.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.576+0000 7ffa6aa5f640 5 monclient: authenticate success, global_id 4613 2024-03-29T10:43:50.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.576+0000 7ffa6aa5f640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:50.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.576+0000 7ffa6aa5f640 4 set_mon_vals no callback set 2024-03-29T10:43:50.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.577+0000 7ffa6aa5f640 10 monclient: shutdown 2024-03-29T10:43:50.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.577+0000 7ffa6aa5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:50.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.577+0000 7ffa6aa5f640 1 -- 172.21.15.84:0/3653466636 >> v1:172.21.15.84:6790/0 conn(0x7ffa5c006a70 legacy=0x7ffa5c0c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:50.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.577+0000 7ffa6aa5f640 1 -- 172.21.15.84:0/3653466636 shutdown_connections 2024-03-29T10:43:50.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.577+0000 7ffa6aa5f640 1 -- 172.21.15.84:0/3653466636 >> 172.21.15.84:0/3653466636 conn(0x7ffa5c0058f0 msgr2=0x7ffa5c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:50.565 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.579+0000 7ffa6aa5f640 1 -- 172.21.15.84:0/3653466636 shutdown_connections 2024-03-29T10:43:50.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.579+0000 7ffa6aa5f640 1 -- 172.21.15.84:0/3653466636 wait complete. 2024-03-29T10:43:50.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.579+0000 7ffa6aa5f640 10 monclient: build_initial_monmap 2024-03-29T10:43:50.567 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.580+0000 7ffa6aa5f640 10 monclient: monmap: 2024-03-29T10:43:50.568 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:43:50.568 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:50.568 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:43:50.568 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:43:50.568 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:50.568 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:50.568 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:43:50.568 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:43:50.568 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:43:50.568 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:50.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.580+0000 7ffa6aa5f640 1 librados: starting msgr at 2024-03-29T10:43:50.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.580+0000 7ffa6aa5f640 1 librados: starting objecter 2024-03-29T10:43:50.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.580+0000 7ffa6aa5f640 1 Processor -- start 2024-03-29T10:43:50.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.580+0000 7ffa6aa5f640 1 -- start start 2024-03-29T10:43:50.568 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.580+0000 7ffa6aa5f640 1 librados: setting wanted keys 2024-03-29T10:43:50.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.580+0000 7ffa6aa5f640 1 librados: calling monclient init 2024-03-29T10:43:50.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.580+0000 7ffa6aa5f640 10 monclient: init 2024-03-29T10:43:50.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.581+0000 7ffa6aa5f640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:50.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.581+0000 7ffa6aa5f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:43:50.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.581+0000 7ffa6aa5f640 10 monclient(hunting): picked mon.noname-c con 0x7ffa5c006a70 addr v1:172.21.15.84:6790/0 2024-03-29T10:43:50.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.581+0000 7ffa6aa5f640 10 monclient(hunting): picked mon.noname-a con 0x7ffa5c029c30 addr v1:172.21.15.84:6789/0 2024-03-29T10:43:50.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.581+0000 7ffa6aa5f640 10 monclient(hunting): picked mon.noname-b con 0x7ffa5c1616f0 addr v1:172.21.15.86:6789/0 2024-03-29T10:43:50.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.583+0000 7ffa6aa5f640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa5c02f190 con 0x7ffa5c029c30 2024-03-29T10:43:50.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.583+0000 7ffa6aa5f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa5c0caad0 con 0x7ffa5c1616f0 2024-03-29T10:43:50.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.583+0000 7ffa6aa5f640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa5c02f400 con 0x7ffa5c006a70 2024-03-29T10:43:50.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.583+0000 7ffa6aa5f640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:50.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.583+0000 7ffa6aa5f640 10 monclient(hunting): authenticate will time out at 1332.846576s 2024-03-29T10:43:50.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.584+0000 7ffa6925c640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7ffa5c029c30 0x7ffa5c031b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:43523/0 (socket says 172.21.15.84:43523) 2024-03-29T10:43:50.569 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.584+0000 7ffa6925c640 1 -- 172.21.15.84:0/4049755143 learned_addr learned my addr 172.21.15.84:0/4049755143 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:43:50.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.584+0000 7ffa5a7fc640 10 client.?.objecter ms_handle_connect 0x7ffa5c029c30 2024-03-29T10:43:50.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.584+0000 7ffa5a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:50.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 1 -- 172.21.15.84:0/4049755143 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1227305184 0 0) 0x7ffa5c02f190 con 0x7ffa5c029c30 2024-03-29T10:43:50.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:43:50.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:50.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:50.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 10 monclient(hunting): my global_id is 4521 2024-03-29T10:43:50.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 1 -- 172.21.15.84:0/4049755143 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa48004eb0 con 0x7ffa5c029c30 2024-03-29T10:43:50.570 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 10 client.?.objecter ms_handle_connect 0x7ffa5c1616f0 2024-03-29T10:43:50.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:50.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 1 -- 172.21.15.84:0/4049755143 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1438378213 0 0) 0x7ffa48004eb0 con 0x7ffa5c029c30 2024-03-29T10:43:50.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 1 -- 172.21.15.84:0/4049755143 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa5c02f190 con 0x7ffa5c029c30 2024-03-29T10:43:50.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 1 -- 172.21.15.84:0/4049755143 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7ffa6404fdd0 con 0x7ffa5c029c30 2024-03-29T10:43:50.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:50.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:50.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 10 monclient(hunting): dump: 2024-03-29T10:43:50.571 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:50.571 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:50.571 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:50.571 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:50.571 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:50.571 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:50.571 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:50.571 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:50.572 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:50.572 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:50.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:50.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 1 -- 172.21.15.84:0/4049755143 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3635146265 0 0) 0x7ffa5c0caad0 con 0x7ffa5c1616f0 2024-03-29T10:43:50.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:43:50.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:50.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:50.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 10 monclient(hunting): my global_id is 4486 2024-03-29T10:43:50.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 1 -- 172.21.15.84:0/4049755143 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa48004eb0 con 0x7ffa5c1616f0 2024-03-29T10:43:50.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 1 -- 172.21.15.84:0/4049755143 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 153210050 0 0) 0x7ffa5c02f190 con 0x7ffa5c029c30 2024-03-29T10:43:50.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 1 -- 172.21.15.84:0/4049755143 >> v1:172.21.15.84:6790/0 conn(0x7ffa5c006a70 legacy=0x7ffa5c02d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:50.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 1 -- 172.21.15.84:0/4049755143 >> v1:172.21.15.86:6789/0 conn(0x7ffa5c1616f0 legacy=0x7ffa5c16c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:50.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:43:50.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 1 monclient: found mon.a 2024-03-29T10:43:50.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:50.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:43:50.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 1 -- 172.21.15.84:0/4049755143 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa5c02cd30 con 0x7ffa5c029c30 2024-03-29T10:43:50.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:43:50.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:43:50.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:50.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 10 client.?.objecter ms_handle_connect 0x7ffa5c006a70 2024-03-29T10:43:50.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.585+0000 7ffa5a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:50.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.586+0000 7ffa5a7fc640 1 -- 172.21.15.84:0/4049755143 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa6404f400 con 0x7ffa5c029c30 2024-03-29T10:43:50.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.586+0000 7ffa5a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:50.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.586+0000 7ffa5a7fc640 1 -- 172.21.15.84:0/4049755143 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7ffa64075710 con 0x7ffa5c029c30 2024-03-29T10:43:50.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.586+0000 7ffa5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:50.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.586+0000 7ffa5a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:50.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.586+0000 7ffa5a7fc640 10 monclient: dump: 2024-03-29T10:43:50.573 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:50.574 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:50.574 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:50.574 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:50.574 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:50.574 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:50.574 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:50.574 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:50.574 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:50.574 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:50.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.586+0000 7ffa5bfff640 4 set_mon_vals no callback set 2024-03-29T10:43:50.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.586+0000 7ffa6aa5f640 5 monclient: authenticate success, global_id 4521 2024-03-29T10:43:50.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.586+0000 7ffa6aa5f640 10 monclient: _renew_subs 2024-03-29T10:43:50.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.586+0000 7ffa6aa5f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:43:50.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.586+0000 7ffa6aa5f640 1 -- 172.21.15.84:0/4049755143 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa5c02cf80 con 0x7ffa5c029c30 2024-03-29T10:43:50.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.586+0000 7ffa6aa5f640 10 client.4521.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:50.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.586+0000 7ffa6aa5f640 10 monclient: _renew_subs 2024-03-29T10:43:50.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.586+0000 7ffa6aa5f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:43:50.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.586+0000 7ffa6aa5f640 1 -- 172.21.15.84:0/4049755143 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa5c160e90 con 0x7ffa5c029c30 2024-03-29T10:43:50.575 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.586+0000 7ffa6aa5f640 1 librados: init done 2024-03-29T10:43:50.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.588+0000 7ffa6aa5f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:50.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.588+0000 7ffa6aa5f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:50.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.588+0000 7ffa6aa5f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:43:50.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.588+0000 7ffa6aa5f640 1 -- 172.21.15.84:0/4049755143 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa5c02d3c0 con 0x7ffa5c029c30 2024-03-29T10:43:50.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.588+0000 7ffa5a7fc640 1 -- 172.21.15.84:0/4049755143 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7ffa64067bf0 con 0x7ffa5c029c30 2024-03-29T10:43:50.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.589+0000 7ffa5a7fc640 1 -- 172.21.15.84:0/4049755143 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7ffa640ff540 con 0x7ffa5c029c30 2024-03-29T10:43:50.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.589+0000 7ffa5a7fc640 10 client.4521.objecter ms_dispatch 0x7ffa64000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:50.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.589+0000 7ffa5a7fc640 3 client.4521.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:50.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.589+0000 7ffa5a7fc640 3 client.4521.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:50.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.589+0000 7ffa5a7fc640 20 client.4521.objecter dump_active .. 0 homeless 2024-03-29T10:43:50.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.590+0000 7ffa69a5d640 10 monclient: get_auth_request con 0x7ffa4807ec40 auth_method 0 2024-03-29T10:43:50.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.590+0000 7ffa5a7fc640 10 client.4521.objecter ms_handle_connect 0x7ffa4807ec40 2024-03-29T10:43:50.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.598+0000 7ffa5a7fc640 1 -- 172.21.15.84:0/4049755143 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffa5c02d3c0 con 0x7ffa5c029c30 2024-03-29T10:43:50.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.598+0000 7ffa5a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:50.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.598+0000 7ffa5a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:50.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.927+0000 7ffa6aa5f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:50.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.927+0000 7ffa6aa5f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:50.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.927+0000 7ffa6aa5f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:43:50.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.927+0000 7ffa6aa5f640 1 -- 172.21.15.84:0/4049755143 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa5c02d3c0 con 0x7ffa5c029c30 2024-03-29T10:43:50.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.928+0000 7ffa5a7fc640 1 -- 172.21.15.84:0/4049755143 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7ffa5c02d3c0 con 0x7ffa5c029c30 2024-03-29T10:43:50.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.928+0000 7ffa5a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:50.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.928+0000 7ffa5a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:50.912 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:43:50.912 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:50.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.931+0000 7ffa6aa5f640 10 librados: watch_flush enter 2024-03-29T10:43:50.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.931+0000 7ffa6aa5f640 10 librados: watch_flush exit 2024-03-29T10:43:50.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.931+0000 7ffa6aa5f640 20 client.4521.objecter shutdown clearing up homeless session... 2024-03-29T10:43:50.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.931+0000 7ffa6aa5f640 10 client.4521.objecter successfully canceled tick 2024-03-29T10:43:50.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.931+0000 7ffa6aa5f640 1 -- 172.21.15.84:0/4049755143 >> v1:172.21.15.86:6816/3337225337 conn(0x7ffa4807ec40 legacy=0x7ffa48081100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:50.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.931+0000 7ffa6aa5f640 10 monclient: shutdown 2024-03-29T10:43:50.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.931+0000 7ffa6aa5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:50.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.931+0000 7ffa6aa5f640 1 -- 172.21.15.84:0/4049755143 >> v1:172.21.15.84:6789/0 conn(0x7ffa5c029c30 legacy=0x7ffa5c031b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:50.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.931+0000 7ffa6aa5f640 1 -- 172.21.15.84:0/4049755143 shutdown_connections 2024-03-29T10:43:50.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.931+0000 7ffa6aa5f640 1 -- 172.21.15.84:0/4049755143 >> 172.21.15.84:0/4049755143 conn(0x7ffa5c0058f0 msgr2=0x7ffa5c0c3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:50.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.932+0000 7ffa6aa5f640 1 -- 172.21.15.84:0/4049755143 shutdown_connections 2024-03-29T10:43:50.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.932+0000 7ffa6aa5f640 1 -- 172.21.15.84:0/4049755143 wait complete. 2024-03-29T10:43:50.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:50.932+0000 7ffa6aa5f640 1 librados: shutdown 2024-03-29T10:43:50.932 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:43:57.740 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:43:57.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.908+0000 7efc87e9a640 10 monclient: get_monmap_and_config 2024-03-29T10:43:57.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.908+0000 7efc87e9a640 10 monclient: build_initial_monmap 2024-03-29T10:43:57.891 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.908+0000 7efc87e9a640 10 monclient: monmap: 2024-03-29T10:43:57.891 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:43:57.891 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:57.891 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:43:57.909372+0000 2024-03-29T10:43:57.892 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:43:57.909372+0000 2024-03-29T10:43:57.892 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:57.892 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:57.892 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:43:57.892 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:43:57.892 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:43:57.892 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:57.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.909+0000 7efc87e9a640 1 Processor -- start 2024-03-29T10:43:57.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.909+0000 7efc87e9a640 1 -- start start 2024-03-29T10:43:57.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.909+0000 7efc87e9a640 10 monclient: init 2024-03-29T10:43:57.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.910+0000 7efc87e9a640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:57.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.910+0000 7efc87e9a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:43:57.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.910+0000 7efc87e9a640 10 monclient(hunting): picked mon.noname-a con 0x7efc800b1f60 addr v1:172.21.15.84:6789/0 2024-03-29T10:43:57.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.910+0000 7efc87e9a640 10 monclient(hunting): picked mon.noname-c con 0x7efc80071eb0 addr v1:172.21.15.86:6789/0 2024-03-29T10:43:57.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.910+0000 7efc87e9a640 10 monclient(hunting): picked mon.noname-b con 0x7efc80143d10 addr v1:172.21.15.84:6790/0 2024-03-29T10:43:57.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.910+0000 7efc87e9a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc8006ef80 con 0x7efc800b1f60 2024-03-29T10:43:57.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.910+0000 7efc87e9a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc8006f260 con 0x7efc80071eb0 2024-03-29T10:43:57.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.910+0000 7efc87e9a640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc800b40c0 con 0x7efc80143d10 2024-03-29T10:43:57.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.910+0000 7efc87e9a640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:57.893 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.910+0000 7efc87e9a640 10 monclient(hunting): authenticate will time out at 1340.173307s 2024-03-29T10:43:57.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.913+0000 7efc86437640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7efc80143d10 0x7efc80146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:59948/0 (socket says 172.21.15.84:59948) 2024-03-29T10:43:57.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.913+0000 7efc85435640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7efc80071eb0 0x7efc80146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:53853/0 (socket says 172.21.15.84:53853) 2024-03-29T10:43:57.896 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.913+0000 7efc85435640 1 -- 172.21.15.84:0/2534319886 learned_addr learned my addr 172.21.15.84:0/2534319886 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:43:57.902 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 1 -- 172.21.15.84:0/2534319886 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1510541482 0 0) 0x7efc8006ef80 con 0x7efc800b1f60 2024-03-29T10:43:57.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 10 monclient(hunting): _negotiate 2024-03-29T10:43:57.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:57.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:57.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 10 monclient(hunting): my global_id is 4524 2024-03-29T10:43:57.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 1 -- 172.21.15.84:0/2534319886 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc6c003520 con 0x7efc800b1f60 2024-03-29T10:43:57.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 1 -- 172.21.15.84:0/2534319886 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1402487251 0 0) 0x7efc8006f260 con 0x7efc80071eb0 2024-03-29T10:43:57.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 10 monclient(hunting): _negotiate 2024-03-29T10:43:57.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:57.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:57.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 10 monclient(hunting): my global_id is 4489 2024-03-29T10:43:57.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 1 -- 172.21.15.84:0/2534319886 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc8006ef80 con 0x7efc80071eb0 2024-03-29T10:43:57.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 1 -- 172.21.15.84:0/2534319886 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1580172177 0 0) 0x7efc6c003520 con 0x7efc800b1f60 2024-03-29T10:43:57.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 1 -- 172.21.15.84:0/2534319886 >> v1:172.21.15.84:6790/0 conn(0x7efc80143d10 legacy=0x7efc80146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:57.903 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 1 -- 172.21.15.84:0/2534319886 >> v1:172.21.15.86:6789/0 conn(0x7efc80071eb0 legacy=0x7efc80146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:57.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 10 monclient: _finish_hunting 0 2024-03-29T10:43:57.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 1 monclient: found mon.noname-a 2024-03-29T10:43:57.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:57.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T10:43:57.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 1 -- 172.21.15.84:0/2534319886 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc80155d70 con 0x7efc800b1f60 2024-03-29T10:43:57.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 10 monclient: _finish_auth 0 2024-03-29T10:43:57.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 10 monclient: _check_auth_tickets 2024-03-29T10:43:57.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:57.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 1 -- 172.21.15.84:0/2534319886 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7efc70001690 con 0x7efc800b1f60 2024-03-29T10:43:57.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:57.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:43:57.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 10 monclient: dump: 2024-03-29T10:43:57.904 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:57.904 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:57.904 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:57.905 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:57.905 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:57.905 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:57.905 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:57.905 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:57.905 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:57.905 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:57.905 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 1 -- 172.21.15.84:0/2534319886 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc700017f0 con 0x7efc800b1f60 2024-03-29T10:43:57.905 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:57.905 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 1 -- 172.21.15.84:0/2534319886 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7efc70001c40 con 0x7efc800b1f60 2024-03-29T10:43:57.905 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:57.905 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:43:57.905 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.916+0000 7efc84c34640 10 monclient: dump: 2024-03-29T10:43:57.905 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:57.905 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:57.906 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:57.906 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:57.906 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:57.906 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:57.906 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:57.906 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:57.906 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:57.906 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:57.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.917+0000 7efc87e9a640 5 monclient: authenticate success, global_id 4524 2024-03-29T10:43:57.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.917+0000 7efc87e9a640 10 monclient: get_monmap_and_config success 2024-03-29T10:43:57.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.917+0000 7efc87e9a640 4 set_mon_vals no callback set 2024-03-29T10:43:57.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.918+0000 7efc87e9a640 10 monclient: shutdown 2024-03-29T10:43:57.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.918+0000 7efc87e9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:57.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.918+0000 7efc87e9a640 1 -- 172.21.15.84:0/2534319886 >> v1:172.21.15.84:6789/0 conn(0x7efc800b1f60 legacy=0x7efc80143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:57.906 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.919+0000 7efc87e9a640 1 -- 172.21.15.84:0/2534319886 shutdown_connections 2024-03-29T10:43:57.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.919+0000 7efc87e9a640 1 -- 172.21.15.84:0/2534319886 >> 172.21.15.84:0/2534319886 conn(0x7efc80066a00 msgr2=0x7efc80066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:57.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.920+0000 7efc87e9a640 1 -- 172.21.15.84:0/2534319886 shutdown_connections 2024-03-29T10:43:57.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.921+0000 7efc87e9a640 1 -- 172.21.15.84:0/2534319886 wait complete. 2024-03-29T10:43:57.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.922+0000 7efc87e9a640 10 monclient: build_initial_monmap 2024-03-29T10:43:57.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.922+0000 7efc87e9a640 10 monclient: monmap: 2024-03-29T10:43:57.909 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:43:57.909 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:43:57.909 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:43:57.909 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:43:57.909 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:43:57.909 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:57.909 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:43:57.909 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:43:57.909 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:43:57.909 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:57.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.922+0000 7efc87e9a640 1 librados: starting msgr at 2024-03-29T10:43:57.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.922+0000 7efc87e9a640 1 librados: starting objecter 2024-03-29T10:43:57.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.922+0000 7efc87e9a640 1 Processor -- start 2024-03-29T10:43:57.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.922+0000 7efc87e9a640 1 -- start start 2024-03-29T10:43:57.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.922+0000 7efc87e9a640 1 librados: setting wanted keys 2024-03-29T10:43:57.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.922+0000 7efc87e9a640 1 librados: calling monclient init 2024-03-29T10:43:57.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.922+0000 7efc87e9a640 10 monclient: init 2024-03-29T10:43:57.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.923+0000 7efc87e9a640 10 monclient: _reopen_session rank -1 2024-03-29T10:43:57.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.923+0000 7efc87e9a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:43:57.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.923+0000 7efc87e9a640 10 monclient(hunting): picked mon.noname-b con 0x7efc800b1f60 addr v1:172.21.15.86:6789/0 2024-03-29T10:43:57.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.923+0000 7efc87e9a640 10 monclient(hunting): picked mon.noname-a con 0x7efc80143d10 addr v1:172.21.15.84:6789/0 2024-03-29T10:43:57.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.923+0000 7efc87e9a640 10 monclient(hunting): picked mon.noname-c con 0x7efc80070d20 addr v1:172.21.15.84:6790/0 2024-03-29T10:43:57.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.923+0000 7efc87e9a640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc801488f0 con 0x7efc80143d10 2024-03-29T10:43:57.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.923+0000 7efc87e9a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc8013b7d0 con 0x7efc800b1f60 2024-03-29T10:43:57.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.923+0000 7efc87e9a640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc8014b0d0 con 0x7efc80070d20 2024-03-29T10:43:57.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.923+0000 7efc87e9a640 10 monclient(hunting): _renew_subs 2024-03-29T10:43:57.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.923+0000 7efc87e9a640 10 monclient(hunting): authenticate will time out at 1340.186924s 2024-03-29T10:43:57.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.927+0000 7efc86437640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7efc80070d20 0x7efc8013b0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:38898/0 (socket says 172.21.15.84:38898) 2024-03-29T10:43:57.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.927+0000 7efc86437640 1 -- 172.21.15.84:0/3210701167 learned_addr learned my addr 172.21.15.84:0/3210701167 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:43:57.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.927+0000 7efc767fc640 10 client.?.objecter ms_handle_connect 0x7efc80070d20 2024-03-29T10:43:57.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.927+0000 7efc767fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:57.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.927+0000 7efc767fc640 10 client.?.objecter ms_handle_connect 0x7efc800b1f60 2024-03-29T10:43:57.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.927+0000 7efc767fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:57.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.928+0000 7efc767fc640 10 client.?.objecter ms_handle_connect 0x7efc80143d10 2024-03-29T10:43:57.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.928+0000 7efc767fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:43:57.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.928+0000 7efc767fc640 1 -- 172.21.15.84:0/3210701167 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3693403595 0 0) 0x7efc8014b0d0 con 0x7efc80070d20 2024-03-29T10:43:57.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.928+0000 7efc767fc640 10 monclient(hunting): _negotiate 2024-03-29T10:43:57.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.928+0000 7efc767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:57.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.928+0000 7efc767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:57.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.928+0000 7efc767fc640 10 monclient(hunting): my global_id is 4622 2024-03-29T10:43:57.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.928+0000 7efc767fc640 1 -- 172.21.15.84:0/3210701167 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc64004eb0 con 0x7efc80070d20 2024-03-29T10:43:57.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.928+0000 7efc767fc640 1 -- 172.21.15.84:0/3210701167 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040759430 0 0) 0x7efc8013b7d0 con 0x7efc800b1f60 2024-03-29T10:43:57.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.928+0000 7efc767fc640 10 monclient(hunting): _negotiate 2024-03-29T10:43:57.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.928+0000 7efc767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:57.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.928+0000 7efc767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:57.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.928+0000 7efc767fc640 10 monclient(hunting): my global_id is 4492 2024-03-29T10:43:57.913 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.928+0000 7efc767fc640 1 -- 172.21.15.84:0/3210701167 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc8014b0d0 con 0x7efc800b1f60 2024-03-29T10:43:57.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.929+0000 7efc767fc640 1 -- 172.21.15.84:0/3210701167 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1135764891 0 0) 0x7efc64004eb0 con 0x7efc80070d20 2024-03-29T10:43:57.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.929+0000 7efc767fc640 1 -- 172.21.15.84:0/3210701167 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc8013b7d0 con 0x7efc80070d20 2024-03-29T10:43:57.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.929+0000 7efc767fc640 1 -- 172.21.15.84:0/3210701167 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1165328834 0 0) 0x7efc801488f0 con 0x7efc80143d10 2024-03-29T10:43:57.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.929+0000 7efc767fc640 10 monclient(hunting): _negotiate 2024-03-29T10:43:57.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.929+0000 7efc767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:43:57.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.929+0000 7efc767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:43:57.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.929+0000 7efc767fc640 10 monclient(hunting): my global_id is 4527 2024-03-29T10:43:57.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.929+0000 7efc767fc640 1 -- 172.21.15.84:0/3210701167 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc64004eb0 con 0x7efc80143d10 2024-03-29T10:43:57.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.929+0000 7efc767fc640 1 -- 172.21.15.84:0/3210701167 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7efc78001c30 con 0x7efc80070d20 2024-03-29T10:43:57.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.929+0000 7efc767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:57.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.929+0000 7efc767fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:43:57.914 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.929+0000 7efc767fc640 10 monclient(hunting): dump: 2024-03-29T10:43:57.915 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:57.915 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:57.915 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:57.915 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:57.915 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:57.915 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:57.915 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:57.915 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:57.915 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:57.915 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:57.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.929+0000 7efc767fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:43:57.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.929+0000 7efc767fc640 1 -- 172.21.15.84:0/3210701167 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3911398432 0 0) 0x7efc8013b7d0 con 0x7efc80070d20 2024-03-29T10:43:57.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.929+0000 7efc767fc640 1 -- 172.21.15.84:0/3210701167 >> v1:172.21.15.86:6789/0 conn(0x7efc800b1f60 legacy=0x7efc801473d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:57.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.929+0000 7efc767fc640 1 -- 172.21.15.84:0/3210701167 >> v1:172.21.15.84:6789/0 conn(0x7efc80143d10 legacy=0x7efc8013a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:57.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.929+0000 7efc767fc640 10 monclient: _finish_hunting 0 2024-03-29T10:43:57.915 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.929+0000 7efc767fc640 1 monclient: found mon.c 2024-03-29T10:43:57.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.929+0000 7efc767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:43:57.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.929+0000 7efc767fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:43:57.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.929+0000 7efc767fc640 1 -- 172.21.15.84:0/3210701167 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc8014b3b0 con 0x7efc80070d20 2024-03-29T10:43:57.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.929+0000 7efc767fc640 10 monclient: _finish_auth 0 2024-03-29T10:43:57.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.929+0000 7efc767fc640 10 monclient: _check_auth_tickets 2024-03-29T10:43:57.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.929+0000 7efc767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:43:57.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.930+0000 7efc767fc640 1 -- 172.21.15.84:0/3210701167 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc78004b60 con 0x7efc80070d20 2024-03-29T10:43:57.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.930+0000 7efc767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:43:57.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.930+0000 7efc767fc640 1 -- 172.21.15.84:0/3210701167 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7efc78004e70 con 0x7efc80070d20 2024-03-29T10:43:57.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.930+0000 7efc767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:43:57.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.930+0000 7efc767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:43:57.916 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.930+0000 7efc767fc640 10 monclient: dump: 2024-03-29T10:43:57.916 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:43:57.916 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:43:57.916 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:57.916 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:43:57.917 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:43:57.917 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:43:57.917 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:43:57.917 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:43:57.917 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:43:57.917 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:43:57.917 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.930+0000 7efc777fe640 4 set_mon_vals no callback set 2024-03-29T10:43:57.918 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.931+0000 7efc87e9a640 5 monclient: authenticate success, global_id 4622 2024-03-29T10:43:57.918 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.931+0000 7efc87e9a640 10 monclient: _renew_subs 2024-03-29T10:43:57.918 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.931+0000 7efc87e9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:43:57.918 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.931+0000 7efc87e9a640 1 -- 172.21.15.84:0/3210701167 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc801491f0 con 0x7efc80070d20 2024-03-29T10:43:57.918 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.931+0000 7efc87e9a640 10 client.4622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:43:57.918 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.931+0000 7efc87e9a640 10 monclient: _renew_subs 2024-03-29T10:43:57.918 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.931+0000 7efc87e9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:43:57.918 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.931+0000 7efc87e9a640 1 -- 172.21.15.84:0/3210701167 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc801496a0 con 0x7efc80070d20 2024-03-29T10:43:57.918 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.931+0000 7efc87e9a640 1 librados: init done 2024-03-29T10:43:57.918 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.934+0000 7efc87e9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:57.918 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.934+0000 7efc87e9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:57.918 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.934+0000 7efc87e9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:43:57.918 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.934+0000 7efc87e9a640 1 -- 172.21.15.84:0/3210701167 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc80075f10 con 0x7efc80070d20 2024-03-29T10:43:57.919 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.934+0000 7efc767fc640 1 -- 172.21.15.84:0/3210701167 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7efc7801ae40 con 0x7efc80070d20 2024-03-29T10:43:57.919 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.935+0000 7efc767fc640 1 -- 172.21.15.84:0/3210701167 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7efc78084e50 con 0x7efc80070d20 2024-03-29T10:43:57.919 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.935+0000 7efc767fc640 10 client.4622.objecter ms_dispatch 0x7efc80000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:43:57.919 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.935+0000 7efc767fc640 3 client.4622.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:43:57.919 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.935+0000 7efc767fc640 3 client.4622.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:43:57.919 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.935+0000 7efc767fc640 20 client.4622.objecter dump_active .. 0 homeless 2024-03-29T10:43:57.919 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.935+0000 7efc85c36640 10 monclient: get_auth_request con 0x7efc6406ded0 auth_method 0 2024-03-29T10:43:57.919 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.936+0000 7efc767fc640 10 client.4622.objecter ms_handle_connect 0x7efc6406ded0 2024-03-29T10:43:57.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.942+0000 7efc767fc640 1 -- 172.21.15.84:0/3210701167 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efc78050560 con 0x7efc80070d20 2024-03-29T10:43:57.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.942+0000 7efc767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:43:57.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:57.942+0000 7efc767fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:43:58.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:58.276+0000 7efc87e9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:58.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:58.276+0000 7efc87e9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:58.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:58.276+0000 7efc87e9a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:43:58.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:58.276+0000 7efc87e9a640 1 -- 172.21.15.84:0/3210701167 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc80149a10 con 0x7efc80070d20 2024-03-29T10:43:58.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:58.277+0000 7efc767fc640 1 -- 172.21.15.84:0/3210701167 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7efc78052660 con 0x7efc80070d20 2024-03-29T10:43:58.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:58.277+0000 7efc767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:43:58.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:58.277+0000 7efc767fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:43:58.261 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:43:58.261 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:43:58.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:58.281+0000 7efc87e9a640 10 librados: watch_flush enter 2024-03-29T10:43:58.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:58.281+0000 7efc87e9a640 10 librados: watch_flush exit 2024-03-29T10:43:58.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:58.281+0000 7efc87e9a640 20 client.4622.objecter shutdown clearing up homeless session... 2024-03-29T10:43:58.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:58.281+0000 7efc87e9a640 10 client.4622.objecter successfully canceled tick 2024-03-29T10:43:58.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:58.281+0000 7efc87e9a640 1 -- 172.21.15.84:0/3210701167 >> v1:172.21.15.86:6816/3337225337 conn(0x7efc6406ded0 legacy=0x7efc64070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:58.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:58.281+0000 7efc87e9a640 10 monclient: shutdown 2024-03-29T10:43:58.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:58.281+0000 7efc87e9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:43:58.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:58.281+0000 7efc87e9a640 1 -- 172.21.15.84:0/3210701167 >> v1:172.21.15.84:6790/0 conn(0x7efc80070d20 legacy=0x7efc8013b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:43:58.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:58.281+0000 7efc87e9a640 1 -- 172.21.15.84:0/3210701167 shutdown_connections 2024-03-29T10:43:58.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:58.281+0000 7efc87e9a640 1 -- 172.21.15.84:0/3210701167 >> 172.21.15.84:0/3210701167 conn(0x7efc80066a00 msgr2=0x7efc800b7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:43:58.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:58.281+0000 7efc87e9a640 1 -- 172.21.15.84:0/3210701167 shutdown_connections 2024-03-29T10:43:58.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:58.281+0000 7efc87e9a640 1 -- 172.21.15.84:0/3210701167 wait complete. 2024-03-29T10:43:58.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:43:58.281+0000 7efc87e9a640 1 librados: shutdown 2024-03-29T10:43:58.283 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:05.091 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:05.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.258+0000 7f5708825640 10 monclient: get_monmap_and_config 2024-03-29T10:44:05.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.258+0000 7f5708825640 10 monclient: build_initial_monmap 2024-03-29T10:44:05.241 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.258+0000 7f5708825640 10 monclient: monmap: 2024-03-29T10:44:05.241 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:44:05.241 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:05.241 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:44:05.259145+0000 2024-03-29T10:44:05.241 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:44:05.259145+0000 2024-03-29T10:44:05.241 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:05.242 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:05.242 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:44:05.242 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:44:05.242 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:44:05.242 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:05.243 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.261+0000 7f5708825640 1 Processor -- start 2024-03-29T10:44:05.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.261+0000 7f5708825640 1 -- start start 2024-03-29T10:44:05.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.261+0000 7f5708825640 10 monclient: init 2024-03-29T10:44:05.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.262+0000 7f5708825640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:05.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.262+0000 7f5708825640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:44:05.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.262+0000 7f5708825640 10 monclient(hunting): picked mon.noname-b con 0x7f57041422b0 addr v1:172.21.15.84:6790/0 2024-03-29T10:44:05.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.262+0000 7f5708825640 10 monclient(hunting): picked mon.noname-c con 0x7f5704142690 addr v1:172.21.15.86:6789/0 2024-03-29T10:44:05.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.262+0000 7f5708825640 10 monclient(hunting): picked mon.noname-a con 0x7f57040b17d0 addr v1:172.21.15.84:6789/0 2024-03-29T10:44:05.245 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.262+0000 7f5708825640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5704142c80 con 0x7f57040b17d0 2024-03-29T10:44:05.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.262+0000 7f5708825640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57040aae40 con 0x7f5704142690 2024-03-29T10:44:05.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.262+0000 7f5708825640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f570413fbd0 con 0x7f57041422b0 2024-03-29T10:44:05.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.262+0000 7f5708825640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:05.246 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.262+0000 7f5708825640 10 monclient(hunting): authenticate will time out at 1347.525514s 2024-03-29T10:44:05.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.266+0000 7f570259c640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f57041422b0 0x7f570413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:54966/0 (socket says 172.21.15.84:54966) 2024-03-29T10:44:05.249 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.266+0000 7f570259c640 1 -- 172.21.15.84:0/2130898867 learned_addr learned my addr 172.21.15.84:0/2130898867 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:44:05.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.267+0000 7f570159a640 1 -- 172.21.15.84:0/2130898867 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596490143 0 0) 0x7f570413fbd0 con 0x7f57041422b0 2024-03-29T10:44:05.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.267+0000 7f570159a640 10 monclient(hunting): _negotiate 2024-03-29T10:44:05.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.267+0000 7f570159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:05.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.267+0000 7f570159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:05.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.267+0000 7f570159a640 10 monclient(hunting): my global_id is 4625 2024-03-29T10:44:05.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.267+0000 7f570159a640 1 -- 172.21.15.84:0/2130898867 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56ec003520 con 0x7f57041422b0 2024-03-29T10:44:05.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.267+0000 7f570159a640 1 -- 172.21.15.84:0/2130898867 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 997179120 0 0) 0x7f5704142c80 con 0x7f57040b17d0 2024-03-29T10:44:05.251 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.267+0000 7f570159a640 10 monclient(hunting): _negotiate 2024-03-29T10:44:05.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.267+0000 7f570159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:05.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.267+0000 7f570159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:05.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.267+0000 7f570159a640 10 monclient(hunting): my global_id is 4530 2024-03-29T10:44:05.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.267+0000 7f570159a640 1 -- 172.21.15.84:0/2130898867 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f570413fbd0 con 0x7f57040b17d0 2024-03-29T10:44:05.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.267+0000 7f570159a640 1 -- 172.21.15.84:0/2130898867 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3258273643 0 0) 0x7f57040aae40 con 0x7f5704142690 2024-03-29T10:44:05.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.267+0000 7f570159a640 10 monclient(hunting): _negotiate 2024-03-29T10:44:05.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.267+0000 7f570159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:05.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.267+0000 7f570159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:05.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.267+0000 7f570159a640 10 monclient(hunting): my global_id is 4495 2024-03-29T10:44:05.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.267+0000 7f570159a640 1 -- 172.21.15.84:0/2130898867 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5704142c80 con 0x7f5704142690 2024-03-29T10:44:05.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.267+0000 7f570159a640 1 -- 172.21.15.84:0/2130898867 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1277634311 0 0) 0x7f56ec003520 con 0x7f57041422b0 2024-03-29T10:44:05.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.267+0000 7f570159a640 1 -- 172.21.15.84:0/2130898867 >> v1:172.21.15.86:6789/0 conn(0x7f5704142690 legacy=0x7f57040b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:05.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.267+0000 7f570159a640 1 -- 172.21.15.84:0/2130898867 >> v1:172.21.15.84:6789/0 conn(0x7f57040b17d0 legacy=0x7f57040b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:05.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.267+0000 7f570159a640 10 monclient: _finish_hunting 0 2024-03-29T10:44:05.252 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.267+0000 7f570159a640 1 monclient: found mon.noname-b 2024-03-29T10:44:05.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.267+0000 7f570159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:05.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.267+0000 7f570159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:44:05.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.267+0000 7f570159a640 1 -- 172.21.15.84:0/2130898867 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5704151940 con 0x7f57041422b0 2024-03-29T10:44:05.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.267+0000 7f570159a640 10 monclient: _finish_auth 0 2024-03-29T10:44:05.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.268+0000 7f570159a640 10 monclient: _check_auth_tickets 2024-03-29T10:44:05.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.268+0000 7f570159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:05.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.268+0000 7f570159a640 1 -- 172.21.15.84:0/2130898867 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f56f4001690 con 0x7f57041422b0 2024-03-29T10:44:05.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.268+0000 7f570159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:05.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.268+0000 7f570159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:05.253 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.268+0000 7f570159a640 10 monclient: dump: 2024-03-29T10:44:05.253 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:05.253 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:05.253 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:05.253 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:05.254 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:05.254 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:05.254 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:05.254 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:05.254 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:05.254 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:05.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.268+0000 7f570159a640 1 -- 172.21.15.84:0/2130898867 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56f40018b0 con 0x7f57041422b0 2024-03-29T10:44:05.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.268+0000 7f570159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:05.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.268+0000 7f570159a640 1 -- 172.21.15.84:0/2130898867 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f56f4001bc0 con 0x7f57041422b0 2024-03-29T10:44:05.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.268+0000 7f570159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:05.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.268+0000 7f570159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:05.254 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.268+0000 7f570159a640 10 monclient: dump: 2024-03-29T10:44:05.254 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:05.254 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:05.255 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:05.255 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:05.255 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:05.255 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:05.255 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:05.255 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:05.255 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:05.255 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:05.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.268+0000 7f5708825640 5 monclient: authenticate success, global_id 4625 2024-03-29T10:44:05.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.268+0000 7f5708825640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:05.255 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.268+0000 7f5708825640 4 set_mon_vals no callback set 2024-03-29T10:44:05.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.269+0000 7f5708825640 10 monclient: shutdown 2024-03-29T10:44:05.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.269+0000 7f5708825640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:05.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.269+0000 7f5708825640 1 -- 172.21.15.84:0/2130898867 >> v1:172.21.15.84:6790/0 conn(0x7f57041422b0 legacy=0x7f570413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:05.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.271+0000 7f5708825640 1 -- 172.21.15.84:0/2130898867 shutdown_connections 2024-03-29T10:44:05.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.271+0000 7f5708825640 1 -- 172.21.15.84:0/2130898867 >> 172.21.15.84:0/2130898867 conn(0x7f5704066a00 msgr2=0x7f5704066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:05.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.271+0000 7f5708825640 1 -- 172.21.15.84:0/2130898867 shutdown_connections 2024-03-29T10:44:05.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.271+0000 7f5708825640 1 -- 172.21.15.84:0/2130898867 wait complete. 2024-03-29T10:44:05.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.272+0000 7f5708825640 10 monclient: build_initial_monmap 2024-03-29T10:44:05.258 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.272+0000 7f5708825640 10 monclient: monmap: 2024-03-29T10:44:05.258 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:44:05.258 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:05.258 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:44:05.258 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:44:05.259 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:05.259 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:05.259 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:44:05.259 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:44:05.259 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:44:05.259 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:05.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.272+0000 7f5708825640 1 librados: starting msgr at 2024-03-29T10:44:05.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.272+0000 7f5708825640 1 librados: starting objecter 2024-03-29T10:44:05.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.273+0000 7f5708825640 1 Processor -- start 2024-03-29T10:44:05.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.273+0000 7f5708825640 1 -- start start 2024-03-29T10:44:05.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.273+0000 7f5708825640 1 librados: setting wanted keys 2024-03-29T10:44:05.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.273+0000 7f5708825640 1 librados: calling monclient init 2024-03-29T10:44:05.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.273+0000 7f5708825640 10 monclient: init 2024-03-29T10:44:05.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.274+0000 7f5708825640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:05.259 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.274+0000 7f5708825640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:44:05.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.274+0000 7f5708825640 10 monclient(hunting): picked mon.noname-c con 0x7f57040b17d0 addr v1:172.21.15.84:6790/0 2024-03-29T10:44:05.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.274+0000 7f5708825640 10 monclient(hunting): picked mon.noname-a con 0x7f57041422b0 addr v1:172.21.15.84:6789/0 2024-03-29T10:44:05.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.274+0000 7f5708825640 10 monclient(hunting): picked mon.noname-b con 0x7f5704142690 addr v1:172.21.15.86:6789/0 2024-03-29T10:44:05.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.274+0000 7f5708825640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5704131370 con 0x7f57041422b0 2024-03-29T10:44:05.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.274+0000 7f5708825640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5704151120 con 0x7f5704142690 2024-03-29T10:44:05.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.274+0000 7f5708825640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5704150570 con 0x7f57040b17d0 2024-03-29T10:44:05.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.274+0000 7f5708825640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:05.260 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.275+0000 7f5708825640 10 monclient(hunting): authenticate will time out at 1347.538129s 2024-03-29T10:44:05.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.276+0000 7f570259c640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f57040b17d0 0x7f570412bb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:59484/0 (socket says 172.21.15.84:59484) 2024-03-29T10:44:05.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.276+0000 7f570259c640 1 -- 172.21.15.84:0/3072908075 learned_addr learned my addr 172.21.15.84:0/3072908075 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:44:05.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.276+0000 7f56f2ffd640 10 client.?.objecter ms_handle_connect 0x7f57040b17d0 2024-03-29T10:44:05.261 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.276+0000 7f56f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:05.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.279+0000 7f56f2ffd640 1 -- 172.21.15.84:0/3072908075 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832792897 0 0) 0x7f5704150570 con 0x7f57040b17d0 2024-03-29T10:44:05.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.279+0000 7f56f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:05.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.279+0000 7f56f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:05.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.279+0000 7f56f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:05.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.279+0000 7f56f2ffd640 10 monclient(hunting): my global_id is 4628 2024-03-29T10:44:05.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.279+0000 7f56f2ffd640 1 -- 172.21.15.84:0/3072908075 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56e4004eb0 con 0x7f57040b17d0 2024-03-29T10:44:05.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.279+0000 7f56f2ffd640 10 client.?.objecter ms_handle_connect 0x7f57041422b0 2024-03-29T10:44:05.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.279+0000 7f56f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:05.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.279+0000 7f56f2ffd640 10 client.?.objecter ms_handle_connect 0x7f5704142690 2024-03-29T10:44:05.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.279+0000 7f56f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:05.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.279+0000 7f56f2ffd640 1 -- 172.21.15.84:0/3072908075 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 547526313 0 0) 0x7f56e4004eb0 con 0x7f57040b17d0 2024-03-29T10:44:05.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.279+0000 7f56f2ffd640 1 -- 172.21.15.84:0/3072908075 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5704150570 con 0x7f57040b17d0 2024-03-29T10:44:05.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.279+0000 7f56f2ffd640 1 -- 172.21.15.84:0/3072908075 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f56f4001610 con 0x7f57040b17d0 2024-03-29T10:44:05.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.279+0000 7f56f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:05.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.279+0000 7f56f2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:05.267 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.279+0000 7f56f2ffd640 10 monclient(hunting): dump: 2024-03-29T10:44:05.267 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:05.267 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:05.267 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:05.268 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:05.268 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:05.268 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:05.268 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:05.268 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:05.268 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:05.268 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:05.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.279+0000 7f56f2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:05.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f56f2ffd640 1 -- 172.21.15.84:0/3072908075 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2797112084 0 0) 0x7f5704131370 con 0x7f57041422b0 2024-03-29T10:44:05.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f56f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:05.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f56f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:05.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f56f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:05.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f56f2ffd640 10 monclient(hunting): my global_id is 4533 2024-03-29T10:44:05.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f56f2ffd640 1 -- 172.21.15.84:0/3072908075 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56e40055e0 con 0x7f57041422b0 2024-03-29T10:44:05.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f56f2ffd640 1 -- 172.21.15.84:0/3072908075 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 130613756 0 0) 0x7f5704150570 con 0x7f57040b17d0 2024-03-29T10:44:05.268 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f56f2ffd640 1 -- 172.21.15.84:0/3072908075 >> v1:172.21.15.86:6789/0 conn(0x7f5704142690 legacy=0x7f57041e7360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:05.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f56f2ffd640 1 -- 172.21.15.84:0/3072908075 >> v1:172.21.15.84:6789/0 conn(0x7f57041422b0 legacy=0x7f57041df020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:05.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f56f2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:44:05.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f56f2ffd640 1 monclient: found mon.c 2024-03-29T10:44:05.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f56f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:05.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f56f2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:44:05.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f56f2ffd640 1 -- 172.21.15.84:0/3072908075 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5704150700 con 0x7f57040b17d0 2024-03-29T10:44:05.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f56f2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:44:05.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f56f2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:44:05.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f56f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:05.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f56f2ffd640 1 -- 172.21.15.84:0/3072908075 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56f4004a90 con 0x7f57040b17d0 2024-03-29T10:44:05.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f56f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:05.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f5700d99640 4 set_mon_vals no callback set 2024-03-29T10:44:05.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f56f2ffd640 1 -- 172.21.15.84:0/3072908075 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f56f4001610 con 0x7f57040b17d0 2024-03-29T10:44:05.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f56f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:05.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f56f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:05.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f56f2ffd640 10 monclient: dump: 2024-03-29T10:44:05.270 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:05.270 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:05.270 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:05.270 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:05.270 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:05.270 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:05.270 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:05.270 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:05.270 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:05.270 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:05.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f5708825640 5 monclient: authenticate success, global_id 4628 2024-03-29T10:44:05.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f5708825640 10 monclient: _renew_subs 2024-03-29T10:44:05.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f5708825640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:44:05.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f5708825640 1 -- 172.21.15.84:0/3072908075 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f570413bd10 con 0x7f57040b17d0 2024-03-29T10:44:05.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f5708825640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:05.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f5708825640 10 monclient: _renew_subs 2024-03-29T10:44:05.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f5708825640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:44:05.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f5708825640 1 -- 172.21.15.84:0/3072908075 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f570412c470 con 0x7f57040b17d0 2024-03-29T10:44:05.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.280+0000 7f5708825640 1 librados: init done 2024-03-29T10:44:05.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.282+0000 7f56f0ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:05.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.282+0000 7f56f0ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:05.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.282+0000 7f56f0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:44:05.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.282+0000 7f56f0ff9640 1 -- 172.21.15.84:0/3072908075 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56dc005080 con 0x7f57040b17d0 2024-03-29T10:44:05.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.285+0000 7f56f2ffd640 1 -- 172.21.15.84:0/3072908075 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f56f401aaf0 con 0x7f57040b17d0 2024-03-29T10:44:05.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.286+0000 7f56f2ffd640 1 -- 172.21.15.84:0/3072908075 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f56f40847b0 con 0x7f57040b17d0 2024-03-29T10:44:05.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.286+0000 7f56f2ffd640 10 client.4628.objecter ms_dispatch 0x7f5704000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:05.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.286+0000 7f56f2ffd640 3 client.4628.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:05.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.286+0000 7f56f2ffd640 3 client.4628.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:05.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.287+0000 7f56f2ffd640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-29T10:44:05.272 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.287+0000 7f5701d9b640 10 monclient: get_auth_request con 0x7f56e406e1e0 auth_method 0 2024-03-29T10:44:05.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.290+0000 7f56f2ffd640 10 client.4628.objecter ms_handle_connect 0x7f56e406e1e0 2024-03-29T10:44:05.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.295+0000 7f56f2ffd640 1 -- 172.21.15.84:0/3072908075 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f56f404fec0 con 0x7f57040b17d0 2024-03-29T10:44:05.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.295+0000 7f56f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:05.278 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.295+0000 7f56f2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:05.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.611+0000 7f56f0ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:05.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.611+0000 7f56f0ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:05.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.611+0000 7f56f0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:44:05.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.611+0000 7f56f0ff9640 1 -- 172.21.15.84:0/3072908075 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56dc005330 con 0x7f57040b17d0 2024-03-29T10:44:05.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.612+0000 7f56f2ffd640 1 -- 172.21.15.84:0/3072908075 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f56f4001470 con 0x7f57040b17d0 2024-03-29T10:44:05.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.612+0000 7f56f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:05.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.612+0000 7f56f2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:05.596 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:44:05.596 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:05.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.616+0000 7f56f0ff9640 10 librados: watch_flush enter 2024-03-29T10:44:05.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.616+0000 7f56f0ff9640 10 librados: watch_flush exit 2024-03-29T10:44:05.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.616+0000 7f56f0ff9640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-29T10:44:05.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.616+0000 7f56f0ff9640 10 client.4628.objecter successfully canceled tick 2024-03-29T10:44:05.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.616+0000 7f56f0ff9640 1 -- 172.21.15.84:0/3072908075 >> v1:172.21.15.86:6816/3337225337 conn(0x7f56e406e1e0 legacy=0x7f56e40706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:05.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.616+0000 7f56f0ff9640 10 monclient: shutdown 2024-03-29T10:44:05.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.616+0000 7f56f0ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:05.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.616+0000 7f56f0ff9640 1 -- 172.21.15.84:0/3072908075 >> v1:172.21.15.84:6790/0 conn(0x7f57040b17d0 legacy=0x7f570412bb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:05.600 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.617+0000 7f56f0ff9640 1 -- 172.21.15.84:0/3072908075 shutdown_connections 2024-03-29T10:44:05.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.617+0000 7f56f0ff9640 1 -- 172.21.15.84:0/3072908075 >> 172.21.15.84:0/3072908075 conn(0x7f5704066a00 msgr2=0x7f57040b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:05.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.617+0000 7f56f0ff9640 1 -- 172.21.15.84:0/3072908075 shutdown_connections 2024-03-29T10:44:05.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.617+0000 7f56f0ff9640 1 -- 172.21.15.84:0/3072908075 wait complete. 2024-03-29T10:44:05.601 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:05.617+0000 7f56f0ff9640 1 librados: shutdown 2024-03-29T10:44:05.619 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:12.426 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:12.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.588+0000 7f96c11f3640 10 monclient: get_monmap_and_config 2024-03-29T10:44:12.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.588+0000 7f96c11f3640 10 monclient: build_initial_monmap 2024-03-29T10:44:12.571 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.588+0000 7f96c11f3640 10 monclient: monmap: 2024-03-29T10:44:12.571 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:44:12.571 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:12.571 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:44:12.589121+0000 2024-03-29T10:44:12.571 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:44:12.589121+0000 2024-03-29T10:44:12.571 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:12.572 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:12.572 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:44:12.572 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:44:12.572 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:44:12.572 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:12.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.589+0000 7f96c11f3640 1 Processor -- start 2024-03-29T10:44:12.572 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.589+0000 7f96c11f3640 1 -- start start 2024-03-29T10:44:12.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.589+0000 7f96c11f3640 10 monclient: init 2024-03-29T10:44:12.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.590+0000 7f96c11f3640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:12.573 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.590+0000 7f96c11f3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:12.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.590+0000 7f96c11f3640 10 monclient(hunting): picked mon.noname-a con 0x7f96bc0b2800 addr v1:172.21.15.84:6789/0 2024-03-29T10:44:12.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.590+0000 7f96c11f3640 10 monclient(hunting): picked mon.noname-c con 0x7f96bc0a9c00 addr v1:172.21.15.86:6789/0 2024-03-29T10:44:12.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.590+0000 7f96c11f3640 10 monclient(hunting): picked mon.noname-b con 0x7f96bc0b38c0 addr v1:172.21.15.84:6790/0 2024-03-29T10:44:12.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.590+0000 7f96c11f3640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96bc141eb0 con 0x7f96bc0b2800 2024-03-29T10:44:12.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.590+0000 7f96c11f3640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96bc0b22d0 con 0x7f96bc0a9c00 2024-03-29T10:44:12.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.590+0000 7f96c11f3640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96bc142d30 con 0x7f96bc0b38c0 2024-03-29T10:44:12.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.590+0000 7f96c11f3640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:12.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.590+0000 7f96c11f3640 10 monclient(hunting): authenticate will time out at 1354.853862s 2024-03-29T10:44:12.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.591+0000 7f96bbfff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f96bc0b2800 0x7f96bc142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:59918/0 (socket says 172.21.15.84:59918) 2024-03-29T10:44:12.574 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.591+0000 7f96bbfff640 1 -- 172.21.15.84:0/2535707791 learned_addr learned my addr 172.21.15.84:0/2535707791 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:44:12.576 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.591+0000 7f96baffd640 1 -- 172.21.15.84:0/2535707791 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498734894 0 0) 0x7f96bc141eb0 con 0x7f96bc0b2800 2024-03-29T10:44:12.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.591+0000 7f96baffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:12.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.591+0000 7f96baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:12.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.591+0000 7f96baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:12.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.591+0000 7f96baffd640 10 monclient(hunting): my global_id is 4536 2024-03-29T10:44:12.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.591+0000 7f96baffd640 1 -- 172.21.15.84:0/2535707791 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96a4003520 con 0x7f96bc0b2800 2024-03-29T10:44:12.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.591+0000 7f96baffd640 1 -- 172.21.15.84:0/2535707791 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 598493340 0 0) 0x7f96bc0b22d0 con 0x7f96bc0a9c00 2024-03-29T10:44:12.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.591+0000 7f96baffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:12.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.591+0000 7f96baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:12.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.591+0000 7f96baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:12.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.591+0000 7f96baffd640 10 monclient(hunting): my global_id is 4498 2024-03-29T10:44:12.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.591+0000 7f96baffd640 1 -- 172.21.15.84:0/2535707791 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96bc141eb0 con 0x7f96bc0a9c00 2024-03-29T10:44:12.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.592+0000 7f96baffd640 1 -- 172.21.15.84:0/2535707791 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3376175026 0 0) 0x7f96a4003520 con 0x7f96bc0b2800 2024-03-29T10:44:12.577 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.592+0000 7f96baffd640 1 -- 172.21.15.84:0/2535707791 >> v1:172.21.15.84:6790/0 conn(0x7f96bc0b38c0 legacy=0x7f96bc14d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:12.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.592+0000 7f96baffd640 1 -- 172.21.15.84:0/2535707791 >> v1:172.21.15.86:6789/0 conn(0x7f96bc0a9c00 legacy=0x7f96bc13bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:12.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.592+0000 7f96baffd640 10 monclient: _finish_hunting 0 2024-03-29T10:44:12.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.592+0000 7f96baffd640 1 monclient: found mon.noname-a 2024-03-29T10:44:12.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.592+0000 7f96baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:12.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.592+0000 7f96baffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T10:44:12.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.592+0000 7f96baffd640 1 -- 172.21.15.84:0/2535707791 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96bc0aaf90 con 0x7f96bc0b2800 2024-03-29T10:44:12.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.592+0000 7f96baffd640 10 monclient: _finish_auth 0 2024-03-29T10:44:12.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.592+0000 7f96baffd640 10 monclient: _check_auth_tickets 2024-03-29T10:44:12.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.592+0000 7f96baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:12.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.592+0000 7f96baffd640 1 -- 172.21.15.84:0/2535707791 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f96ac001690 con 0x7f96bc0b2800 2024-03-29T10:44:12.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.592+0000 7f96baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:12.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.592+0000 7f96baffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:12.578 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.592+0000 7f96baffd640 10 monclient: dump: 2024-03-29T10:44:12.578 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:12.579 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:12.579 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:12.579 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:12.579 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:12.579 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:12.579 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:12.579 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:12.579 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:12.579 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:12.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.592+0000 7f96c11f3640 5 monclient: authenticate success, global_id 4536 2024-03-29T10:44:12.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.592+0000 7f96c11f3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:12.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.592+0000 7f96baffd640 1 -- 172.21.15.84:0/2535707791 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96ac0017f0 con 0x7f96bc0b2800 2024-03-29T10:44:12.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.592+0000 7f96baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:12.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.592+0000 7f96baffd640 1 -- 172.21.15.84:0/2535707791 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f96ac001b80 con 0x7f96bc0b2800 2024-03-29T10:44:12.579 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.592+0000 7f96baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:12.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.592+0000 7f96baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:12.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.592+0000 7f96baffd640 10 monclient: dump: 2024-03-29T10:44:12.580 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:12.580 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:12.580 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:12.580 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:12.580 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:12.580 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:12.580 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:12.580 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:12.580 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:12.580 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:12.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.592+0000 7f96c11f3640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:12.580 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.592+0000 7f96c11f3640 4 set_mon_vals no callback set 2024-03-29T10:44:12.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.593+0000 7f96c11f3640 10 monclient: shutdown 2024-03-29T10:44:12.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.593+0000 7f96c11f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:12.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.593+0000 7f96c11f3640 1 -- 172.21.15.84:0/2535707791 >> v1:172.21.15.84:6789/0 conn(0x7f96bc0b2800 legacy=0x7f96bc142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:12.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.593+0000 7f96c11f3640 1 -- 172.21.15.84:0/2535707791 shutdown_connections 2024-03-29T10:44:12.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.593+0000 7f96c11f3640 1 -- 172.21.15.84:0/2535707791 >> 172.21.15.84:0/2535707791 conn(0x7f96bc066a00 msgr2=0x7f96bc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:12.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.593+0000 7f96c11f3640 1 -- 172.21.15.84:0/2535707791 shutdown_connections 2024-03-29T10:44:12.581 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.594+0000 7f96c11f3640 1 -- 172.21.15.84:0/2535707791 wait complete. 2024-03-29T10:44:12.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.595+0000 7f96c11f3640 10 monclient: build_initial_monmap 2024-03-29T10:44:12.587 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.595+0000 7f96c11f3640 10 monclient: monmap: 2024-03-29T10:44:12.587 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:44:12.587 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:12.587 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:44:12.587 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:44:12.588 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:12.588 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:12.589 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:44:12.589 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:44:12.589 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:44:12.589 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:12.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.595+0000 7f96c11f3640 1 librados: starting msgr at 2024-03-29T10:44:12.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.595+0000 7f96c11f3640 1 librados: starting objecter 2024-03-29T10:44:12.589 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.595+0000 7f96c11f3640 1 Processor -- start 2024-03-29T10:44:12.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.596+0000 7f96c11f3640 1 -- start start 2024-03-29T10:44:12.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.596+0000 7f96c11f3640 1 librados: setting wanted keys 2024-03-29T10:44:12.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.596+0000 7f96c11f3640 1 librados: calling monclient init 2024-03-29T10:44:12.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.596+0000 7f96c11f3640 10 monclient: init 2024-03-29T10:44:12.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.597+0000 7f96c11f3640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:12.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.597+0000 7f96c11f3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:44:12.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.597+0000 7f96c11f3640 10 monclient(hunting): picked mon.noname-a con 0x7f96bc0a9c00 addr v1:172.21.15.84:6789/0 2024-03-29T10:44:12.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.597+0000 7f96c11f3640 10 monclient(hunting): picked mon.noname-c con 0x7f96bc0b2800 addr v1:172.21.15.84:6790/0 2024-03-29T10:44:12.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.597+0000 7f96bbfff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f96bc0a9c00 0x7f96bc137e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:35958/0 (socket says 172.21.15.84:35958) 2024-03-29T10:44:12.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.597+0000 7f96bbfff640 1 -- 172.21.15.84:0/3935342097 learned_addr learned my addr 172.21.15.84:0/3935342097 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:44:12.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.597+0000 7f96b8ff9640 10 client.?.objecter ms_handle_connect 0x7f96bc0a9c00 2024-03-29T10:44:12.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.597+0000 7f96b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:12.590 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.597+0000 7f96b8ff9640 10 client.?.objecter ms_handle_connect 0x7f96bc0b2800 2024-03-29T10:44:12.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.597+0000 7f96b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:12.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.597+0000 7f96c11f3640 10 monclient(hunting): picked mon.noname-b con 0x7f96bc0b38c0 addr v1:172.21.15.86:6789/0 2024-03-29T10:44:12.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.598+0000 7f96c11f3640 1 -- 172.21.15.84:0/3935342097 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96bc13f380 con 0x7f96bc0a9c00 2024-03-29T10:44:12.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.598+0000 7f96c11f3640 1 -- 172.21.15.84:0/3935342097 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96bc0acd30 con 0x7f96bc0b38c0 2024-03-29T10:44:12.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.598+0000 7f96c11f3640 1 -- 172.21.15.84:0/3935342097 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96bc0ad010 con 0x7f96bc0b2800 2024-03-29T10:44:12.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.598+0000 7f96c11f3640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:12.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.598+0000 7f96b8ff9640 1 -- 172.21.15.84:0/3935342097 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 340718548 0 0) 0x7f96bc13f380 con 0x7f96bc0a9c00 2024-03-29T10:44:12.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.598+0000 7f96c11f3640 10 monclient(hunting): authenticate will time out at 1354.861604s 2024-03-29T10:44:12.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.598+0000 7f96b8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:44:12.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.598+0000 7f96b8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:12.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.598+0000 7f96b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:12.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.598+0000 7f96b8ff9640 10 monclient(hunting): my global_id is 4539 2024-03-29T10:44:12.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.598+0000 7f96b8ff9640 1 -- 172.21.15.84:0/3935342097 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96a8004eb0 con 0x7f96bc0a9c00 2024-03-29T10:44:12.591 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.598+0000 7f96b8ff9640 1 -- 172.21.15.84:0/3935342097 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1046072166 0 0) 0x7f96bc0ad010 con 0x7f96bc0b2800 2024-03-29T10:44:12.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.598+0000 7f96b8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:44:12.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.598+0000 7f96b8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:12.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.598+0000 7f96b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:12.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.598+0000 7f96b8ff9640 10 monclient(hunting): my global_id is 4634 2024-03-29T10:44:12.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.598+0000 7f96b8ff9640 1 -- 172.21.15.84:0/3935342097 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96bc13f380 con 0x7f96bc0b2800 2024-03-29T10:44:12.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.598+0000 7f96b8ff9640 10 client.?.objecter ms_handle_connect 0x7f96bc0b38c0 2024-03-29T10:44:12.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.598+0000 7f96b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:12.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 1 -- 172.21.15.84:0/3935342097 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1830926254 0 0) 0x7f96bc13f380 con 0x7f96bc0b2800 2024-03-29T10:44:12.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 1 -- 172.21.15.84:0/3935342097 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96bc0ad010 con 0x7f96bc0b2800 2024-03-29T10:44:12.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 1 -- 172.21.15.84:0/3935342097 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f96b0001930 con 0x7f96bc0b2800 2024-03-29T10:44:12.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:12.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:12.592 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 10 monclient(hunting): dump: 2024-03-29T10:44:12.592 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:12.592 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:12.593 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:12.593 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:12.593 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:12.593 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:12.593 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:12.593 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:12.593 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:12.593 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:12.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:12.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 1 -- 172.21.15.84:0/3935342097 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3996475032 0 0) 0x7f96a8004eb0 con 0x7f96bc0a9c00 2024-03-29T10:44:12.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 1 -- 172.21.15.84:0/3935342097 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96bc13f380 con 0x7f96bc0a9c00 2024-03-29T10:44:12.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 1 -- 172.21.15.84:0/3935342097 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f96ac001b00 con 0x7f96bc0a9c00 2024-03-29T10:44:12.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:12.593 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:12.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 10 monclient(hunting): dump: 2024-03-29T10:44:12.594 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:12.594 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:12.594 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:12.594 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:12.594 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:12.594 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:12.594 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:12.594 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:12.594 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:12.594 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:12.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 1 -- 172.21.15.84:0/3935342097 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1432874677 0 0) 0x7f96bc0ad010 con 0x7f96bc0b2800 2024-03-29T10:44:12.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 1 -- 172.21.15.84:0/3935342097 >> v1:172.21.15.86:6789/0 conn(0x7f96bc0b38c0 legacy=0x7f96bc1db220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:12.594 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 1 -- 172.21.15.84:0/3935342097 >> v1:172.21.15.84:6789/0 conn(0x7f96bc0a9c00 legacy=0x7f96bc137e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:12.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:44:12.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 1 monclient: found mon.c 2024-03-29T10:44:12.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:12.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:44:12.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 1 -- 172.21.15.84:0/3935342097 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96bc0ab470 con 0x7f96bc0b2800 2024-03-29T10:44:12.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 10 monclient: _finish_auth 0 2024-03-29T10:44:12.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:44:12.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:12.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96c11f3640 5 monclient: authenticate success, global_id 4634 2024-03-29T10:44:12.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96c11f3640 10 monclient: _renew_subs 2024-03-29T10:44:12.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96c11f3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:44:12.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96c11f3640 1 -- 172.21.15.84:0/3935342097 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96bc0ab720 con 0x7f96bc0b2800 2024-03-29T10:44:12.595 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96c11f3640 10 client.4634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:12.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96c11f3640 10 monclient: _renew_subs 2024-03-29T10:44:12.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96c11f3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:44:12.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96c11f3640 1 -- 172.21.15.84:0/3935342097 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96bc138730 con 0x7f96bc0b2800 2024-03-29T10:44:12.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96c11f3640 1 librados: init done 2024-03-29T10:44:12.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 1 -- 172.21.15.84:0/3935342097 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96b0003610 con 0x7f96bc0b2800 2024-03-29T10:44:12.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:12.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 1 -- 172.21.15.84:0/3935342097 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f96b00039e0 con 0x7f96bc0b2800 2024-03-29T10:44:12.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:12.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:12.596 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.599+0000 7f96b8ff9640 10 monclient: dump: 2024-03-29T10:44:12.596 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:12.596 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:12.596 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:12.596 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:12.597 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:12.597 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:12.597 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:12.597 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:12.597 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:12.597 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:12.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.600+0000 7f96ba7fc640 4 set_mon_vals no callback set 2024-03-29T10:44:12.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.601+0000 7f96c11f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:12.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.601+0000 7f96c11f3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:12.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.601+0000 7f96c11f3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:44:12.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.601+0000 7f96c11f3640 1 -- 172.21.15.84:0/3935342097 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f969c005080 con 0x7f96bc0b2800 2024-03-29T10:44:12.597 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.602+0000 7f96b8ff9640 1 -- 172.21.15.84:0/3935342097 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f96b0003b40 con 0x7f96bc0b2800 2024-03-29T10:44:12.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.603+0000 7f96b8ff9640 1 -- 172.21.15.84:0/3935342097 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f96b0083880 con 0x7f96bc0b2800 2024-03-29T10:44:12.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.603+0000 7f96b8ff9640 10 client.4634.objecter ms_dispatch 0x7f96bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:12.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.603+0000 7f96b8ff9640 3 client.4634.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:12.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.603+0000 7f96b8ff9640 3 client.4634.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:12.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.603+0000 7f96b8ff9640 20 client.4634.objecter dump_active .. 0 homeless 2024-03-29T10:44:12.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.603+0000 7f96bbfff640 10 monclient: get_auth_request con 0x7f96a806df20 auth_method 0 2024-03-29T10:44:12.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.603+0000 7f96b8ff9640 10 client.4634.objecter ms_handle_connect 0x7f96a806df20 2024-03-29T10:44:12.598 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.611+0000 7f96b8ff9640 1 -- 172.21.15.84:0/3935342097 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f96b004f160 con 0x7f96bc0b2800 2024-03-29T10:44:12.599 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.611+0000 7f96b8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:12.599 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.611+0000 7f96b8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:12.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.941+0000 7f96c11f3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:12.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.941+0000 7f96c11f3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:12.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.941+0000 7f96c11f3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:44:12.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.941+0000 7f96c11f3640 1 -- 172.21.15.84:0/3935342097 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f969c005330 con 0x7f96bc0b2800 2024-03-29T10:44:12.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.943+0000 7f96b8ff9640 1 -- 172.21.15.84:0/3935342097 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f96b0051090 con 0x7f96bc0b2800 2024-03-29T10:44:12.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.943+0000 7f96b8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:12.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.943+0000 7f96b8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:12.926 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:44:12.926 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:12.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.947+0000 7f96c11f3640 10 librados: watch_flush enter 2024-03-29T10:44:12.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.947+0000 7f96c11f3640 10 librados: watch_flush exit 2024-03-29T10:44:12.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.947+0000 7f96c11f3640 20 client.4634.objecter shutdown clearing up homeless session... 2024-03-29T10:44:12.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.947+0000 7f96c11f3640 10 client.4634.objecter successfully canceled tick 2024-03-29T10:44:12.930 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.947+0000 7f96c11f3640 1 -- 172.21.15.84:0/3935342097 >> v1:172.21.15.86:6816/3337225337 conn(0x7f96a806df20 legacy=0x7f96a80703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:12.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.947+0000 7f96c11f3640 10 monclient: shutdown 2024-03-29T10:44:12.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.947+0000 7f96c11f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:12.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.947+0000 7f96c11f3640 1 -- 172.21.15.84:0/3935342097 >> v1:172.21.15.84:6790/0 conn(0x7f96bc0b2800 legacy=0x7f96bc1dab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:12.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.949+0000 7f96c11f3640 1 -- 172.21.15.84:0/3935342097 shutdown_connections 2024-03-29T10:44:12.931 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.949+0000 7f96c11f3640 1 -- 172.21.15.84:0/3935342097 >> 172.21.15.84:0/3935342097 conn(0x7f96bc066a00 msgr2=0x7f96bc1405b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:12.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.949+0000 7f96c11f3640 1 -- 172.21.15.84:0/3935342097 shutdown_connections 2024-03-29T10:44:12.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.949+0000 7f96c11f3640 1 -- 172.21.15.84:0/3935342097 wait complete. 2024-03-29T10:44:12.932 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:12.949+0000 7f96c11f3640 1 librados: shutdown 2024-03-29T10:44:12.950 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:19.756 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:19.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.921+0000 7f52913cd640 10 monclient: get_monmap_and_config 2024-03-29T10:44:19.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.921+0000 7f52913cd640 10 monclient: build_initial_monmap 2024-03-29T10:44:19.904 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.921+0000 7f52913cd640 10 monclient: monmap: 2024-03-29T10:44:19.904 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:44:19.904 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:19.904 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:44:19.922235+0000 2024-03-29T10:44:19.904 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:44:19.922235+0000 2024-03-29T10:44:19.904 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:19.904 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:19.904 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:44:19.904 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:44:19.905 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:44:19.905 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:19.905 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.922+0000 7f52913cd640 1 Processor -- start 2024-03-29T10:44:19.905 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.922+0000 7f52913cd640 1 -- start start 2024-03-29T10:44:19.905 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.922+0000 7f52913cd640 10 monclient: init 2024-03-29T10:44:19.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.924+0000 7f52913cd640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:19.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.924+0000 7f52913cd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:44:19.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.924+0000 7f52913cd640 10 monclient(hunting): picked mon.noname-c con 0x7f528c1457f0 addr v1:172.21.15.86:6789/0 2024-03-29T10:44:19.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.924+0000 7f52913cd640 10 monclient(hunting): picked mon.noname-b con 0x7f528c13f650 addr v1:172.21.15.84:6790/0 2024-03-29T10:44:19.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.924+0000 7f52913cd640 10 monclient(hunting): picked mon.noname-a con 0x7f528c0a9d70 addr v1:172.21.15.84:6789/0 2024-03-29T10:44:19.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.924+0000 7f52913cd640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f528c0aaef0 con 0x7f528c0a9d70 2024-03-29T10:44:19.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.924+0000 7f52913cd640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f528c145bd0 con 0x7f528c1457f0 2024-03-29T10:44:19.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.924+0000 7f52913cd640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f528c145eb0 con 0x7f528c13f650 2024-03-29T10:44:19.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.924+0000 7f52913cd640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:19.907 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.924+0000 7f52913cd640 10 monclient(hunting): authenticate will time out at 1362.187524s 2024-03-29T10:44:19.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.924+0000 7f528a7fc640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f528c13f650 0x7f528c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:55374/0 (socket says 172.21.15.84:55374) 2024-03-29T10:44:19.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.924+0000 7f528affd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f528c1457f0 0x7f528c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:58862/0 (socket says 172.21.15.84:58862) 2024-03-29T10:44:19.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.924+0000 7f528a7fc640 1 -- 172.21.15.84:0/1524470521 learned_addr learned my addr 172.21.15.84:0/1524470521 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:44:19.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.925+0000 7f5289ffb640 1 -- 172.21.15.84:0/1524470521 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3249199138 0 0) 0x7f528c145eb0 con 0x7f528c13f650 2024-03-29T10:44:19.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.925+0000 7f5289ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:44:19.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.925+0000 7f5289ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:19.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.925+0000 7f5289ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:19.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.925+0000 7f5289ffb640 10 monclient(hunting): my global_id is 4637 2024-03-29T10:44:19.908 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.925+0000 7f5289ffb640 1 -- 172.21.15.84:0/1524470521 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5274003520 con 0x7f528c13f650 2024-03-29T10:44:19.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.925+0000 7f5289ffb640 1 -- 172.21.15.84:0/1524470521 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2039708581 0 0) 0x7f528c0aaef0 con 0x7f528c0a9d70 2024-03-29T10:44:19.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.925+0000 7f5289ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:44:19.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.925+0000 7f5289ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:19.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.925+0000 7f5289ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:19.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.925+0000 7f5289ffb640 10 monclient(hunting): my global_id is 4542 2024-03-29T10:44:19.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.925+0000 7f5289ffb640 1 -- 172.21.15.84:0/1524470521 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f528c145eb0 con 0x7f528c0a9d70 2024-03-29T10:44:19.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.926+0000 7f5289ffb640 1 -- 172.21.15.84:0/1524470521 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3752842402 0 0) 0x7f528c145eb0 con 0x7f528c0a9d70 2024-03-29T10:44:19.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.926+0000 7f5289ffb640 1 -- 172.21.15.84:0/1524470521 >> v1:172.21.15.84:6790/0 conn(0x7f528c13f650 legacy=0x7f528c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:19.909 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.926+0000 7f5289ffb640 1 -- 172.21.15.84:0/1524470521 >> v1:172.21.15.86:6789/0 conn(0x7f528c1457f0 legacy=0x7f528c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:19.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.926+0000 7f5289ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:44:19.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.926+0000 7f5289ffb640 1 monclient: found mon.noname-a 2024-03-29T10:44:19.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.926+0000 7f5289ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:19.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.926+0000 7f5289ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T10:44:19.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.926+0000 7f5289ffb640 1 -- 172.21.15.84:0/1524470521 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528c146340 con 0x7f528c0a9d70 2024-03-29T10:44:19.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.926+0000 7f5289ffb640 10 monclient: _finish_auth 0 2024-03-29T10:44:19.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.926+0000 7f5289ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:44:19.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.926+0000 7f5289ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:19.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.926+0000 7f5289ffb640 1 -- 172.21.15.84:0/1524470521 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f52840019f0 con 0x7f528c0a9d70 2024-03-29T10:44:19.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.926+0000 7f5289ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:19.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.926+0000 7f5289ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:19.910 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.926+0000 7f5289ffb640 10 monclient: dump: 2024-03-29T10:44:19.910 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:19.910 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:19.910 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:19.911 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:19.911 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:19.911 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:19.911 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:19.911 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:19.911 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:19.911 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:19.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.926+0000 7f52913cd640 5 monclient: authenticate success, global_id 4542 2024-03-29T10:44:19.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.926+0000 7f52913cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:19.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.926+0000 7f5289ffb640 1 -- 172.21.15.84:0/1524470521 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5284001b50 con 0x7f528c0a9d70 2024-03-29T10:44:19.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.926+0000 7f5289ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:19.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.926+0000 7f5289ffb640 1 -- 172.21.15.84:0/1524470521 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f5284001fa0 con 0x7f528c0a9d70 2024-03-29T10:44:19.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.926+0000 7f5289ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:19.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.926+0000 7f5289ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:19.911 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.926+0000 7f5289ffb640 10 monclient: dump: 2024-03-29T10:44:19.911 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:19.912 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:19.912 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:19.912 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:19.912 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:19.912 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:19.912 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:19.912 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:19.912 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:19.912 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:19.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.926+0000 7f52913cd640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:19.912 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.926+0000 7f52913cd640 4 set_mon_vals no callback set 2024-03-29T10:44:19.920 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.927+0000 7f52913cd640 10 monclient: shutdown 2024-03-29T10:44:19.920 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.927+0000 7f52913cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:19.920 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.927+0000 7f52913cd640 1 -- 172.21.15.84:0/1524470521 >> v1:172.21.15.84:6789/0 conn(0x7f528c0a9d70 legacy=0x7f528c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:19.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.927+0000 7f52913cd640 1 -- 172.21.15.84:0/1524470521 shutdown_connections 2024-03-29T10:44:19.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.927+0000 7f52913cd640 1 -- 172.21.15.84:0/1524470521 >> 172.21.15.84:0/1524470521 conn(0x7f528c066a00 msgr2=0x7f528c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:19.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.927+0000 7f52913cd640 1 -- 172.21.15.84:0/1524470521 shutdown_connections 2024-03-29T10:44:19.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.927+0000 7f52913cd640 1 -- 172.21.15.84:0/1524470521 wait complete. 2024-03-29T10:44:19.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.928+0000 7f52913cd640 10 monclient: build_initial_monmap 2024-03-29T10:44:19.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.928+0000 7f52913cd640 10 monclient: monmap: 2024-03-29T10:44:19.921 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:44:19.921 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:19.921 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:44:19.921 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:44:19.921 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:19.921 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:19.921 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:44:19.921 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:44:19.921 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:44:19.921 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:19.921 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.929+0000 7f52913cd640 1 librados: starting msgr at 2024-03-29T10:44:19.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.929+0000 7f52913cd640 1 librados: starting objecter 2024-03-29T10:44:19.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.929+0000 7f52913cd640 1 Processor -- start 2024-03-29T10:44:19.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.929+0000 7f52913cd640 1 -- start start 2024-03-29T10:44:19.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.929+0000 7f52913cd640 1 librados: setting wanted keys 2024-03-29T10:44:19.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.929+0000 7f52913cd640 1 librados: calling monclient init 2024-03-29T10:44:19.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.929+0000 7f52913cd640 10 monclient: init 2024-03-29T10:44:19.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.930+0000 7f52913cd640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:19.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.930+0000 7f52913cd640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:44:19.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.930+0000 7f52913cd640 10 monclient(hunting): picked mon.noname-b con 0x7f528c0a9d70 addr v1:172.21.15.86:6789/0 2024-03-29T10:44:19.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.930+0000 7f528affd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f528c0a9d70 0x7f528c0711a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:59506/0 (socket says 172.21.15.84:59506) 2024-03-29T10:44:19.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.931+0000 7f528affd640 1 -- 172.21.15.84:0/840006687 learned_addr learned my addr 172.21.15.84:0/840006687 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:44:19.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.931+0000 7f52913cd640 10 monclient(hunting): picked mon.noname-a con 0x7f528c13f650 addr v1:172.21.15.84:6789/0 2024-03-29T10:44:19.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.931+0000 7f52913cd640 10 monclient(hunting): picked mon.noname-c con 0x7f528c1457f0 addr v1:172.21.15.84:6790/0 2024-03-29T10:44:19.922 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.931+0000 7f52913cd640 1 -- 172.21.15.84:0/840006687 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f528c0ac8b0 con 0x7f528c13f650 2024-03-29T10:44:19.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.931+0000 7f52913cd640 1 -- 172.21.15.84:0/840006687 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f528c071fc0 con 0x7f528c0a9d70 2024-03-29T10:44:19.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.931+0000 7f52737fe640 10 client.?.objecter ms_handle_connect 0x7f528c0a9d70 2024-03-29T10:44:19.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.931+0000 7f52737fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:19.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.931+0000 7f52913cd640 1 -- 172.21.15.84:0/840006687 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f528c0b16f0 con 0x7f528c1457f0 2024-03-29T10:44:19.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.931+0000 7f52913cd640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:19.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.931+0000 7f52913cd640 10 monclient(hunting): authenticate will time out at 1362.194910s 2024-03-29T10:44:19.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.931+0000 7f52737fe640 10 client.?.objecter ms_handle_connect 0x7f528c13f650 2024-03-29T10:44:19.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.932+0000 7f52737fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:19.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.932+0000 7f52737fe640 1 -- 172.21.15.84:0/840006687 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816710304 0 0) 0x7f528c071fc0 con 0x7f528c0a9d70 2024-03-29T10:44:19.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.932+0000 7f52737fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:19.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.932+0000 7f52737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:19.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.932+0000 7f52737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:19.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.932+0000 7f52737fe640 10 monclient(hunting): my global_id is 4507 2024-03-29T10:44:19.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.932+0000 7f52737fe640 1 -- 172.21.15.84:0/840006687 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5278004eb0 con 0x7f528c0a9d70 2024-03-29T10:44:19.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.932+0000 7f52737fe640 10 client.?.objecter ms_handle_connect 0x7f528c1457f0 2024-03-29T10:44:19.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.932+0000 7f52737fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:19.923 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.932+0000 7f52737fe640 1 -- 172.21.15.84:0/840006687 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3931085355 0 0) 0x7f528c0ac8b0 con 0x7f528c13f650 2024-03-29T10:44:19.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.932+0000 7f52737fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:19.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.932+0000 7f52737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:19.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.932+0000 7f52737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:19.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.932+0000 7f52737fe640 10 monclient(hunting): my global_id is 4545 2024-03-29T10:44:19.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.932+0000 7f52737fe640 1 -- 172.21.15.84:0/840006687 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f528c071fc0 con 0x7f528c13f650 2024-03-29T10:44:19.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.932+0000 7f52737fe640 1 -- 172.21.15.84:0/840006687 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1856300182 0 0) 0x7f528c0b16f0 con 0x7f528c1457f0 2024-03-29T10:44:19.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.932+0000 7f52737fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:19.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.932+0000 7f52737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:19.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.932+0000 7f52737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:19.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.932+0000 7f52737fe640 10 monclient(hunting): my global_id is 4640 2024-03-29T10:44:19.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.932+0000 7f52737fe640 1 -- 172.21.15.84:0/840006687 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f528c0ac8b0 con 0x7f528c1457f0 2024-03-29T10:44:19.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.932+0000 7f52737fe640 1 -- 172.21.15.84:0/840006687 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 230417594 0 0) 0x7f5278004eb0 con 0x7f528c0a9d70 2024-03-29T10:44:19.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.932+0000 7f52737fe640 1 -- 172.21.15.84:0/840006687 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f528c0b16f0 con 0x7f528c0a9d70 2024-03-29T10:44:19.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.932+0000 7f52737fe640 1 -- 172.21.15.84:0/840006687 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f527c001850 con 0x7f528c0a9d70 2024-03-29T10:44:19.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.932+0000 7f52737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:19.924 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.932+0000 7f52737fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:19.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.932+0000 7f52737fe640 10 monclient(hunting): dump: 2024-03-29T10:44:19.925 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:19.925 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:19.925 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:19.925 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:19.925 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:19.925 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:19.925 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:19.925 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:19.925 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:19.925 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:19.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.932+0000 7f52737fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:19.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.933+0000 7f52737fe640 1 -- 172.21.15.84:0/840006687 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 360828301 0 0) 0x7f528c0b16f0 con 0x7f528c0a9d70 2024-03-29T10:44:19.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.933+0000 7f52737fe640 1 -- 172.21.15.84:0/840006687 >> v1:172.21.15.84:6790/0 conn(0x7f528c1457f0 legacy=0x7f528c1e6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:19.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.933+0000 7f52737fe640 1 -- 172.21.15.84:0/840006687 >> v1:172.21.15.84:6789/0 conn(0x7f528c13f650 legacy=0x7f528c0718b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:19.925 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.933+0000 7f52737fe640 10 monclient: _finish_hunting 0 2024-03-29T10:44:19.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.933+0000 7f52737fe640 1 monclient: found mon.b 2024-03-29T10:44:19.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.933+0000 7f52737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:19.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.933+0000 7f52737fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:44:19.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.933+0000 7f52737fe640 1 -- 172.21.15.84:0/840006687 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528c0b1880 con 0x7f528c0a9d70 2024-03-29T10:44:19.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.933+0000 7f52737fe640 10 monclient: _finish_auth 0 2024-03-29T10:44:19.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.933+0000 7f52737fe640 10 monclient: _check_auth_tickets 2024-03-29T10:44:19.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.933+0000 7f52737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:19.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.933+0000 7f52913cd640 5 monclient: authenticate success, global_id 4507 2024-03-29T10:44:19.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.933+0000 7f52913cd640 10 monclient: _renew_subs 2024-03-29T10:44:19.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.933+0000 7f52913cd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:44:19.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.933+0000 7f52913cd640 1 -- 172.21.15.84:0/840006687 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f528c13bd80 con 0x7f528c0a9d70 2024-03-29T10:44:19.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.933+0000 7f52913cd640 10 client.4507.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:19.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.933+0000 7f52913cd640 10 monclient: _renew_subs 2024-03-29T10:44:19.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.933+0000 7f52913cd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:44:19.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.933+0000 7f52913cd640 1 -- 172.21.15.84:0/840006687 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f528c0ab7e0 con 0x7f528c0a9d70 2024-03-29T10:44:19.926 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.933+0000 7f52913cd640 1 librados: init done 2024-03-29T10:44:19.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.934+0000 7f52737fe640 1 -- 172.21.15.84:0/840006687 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f527c001d40 con 0x7f528c0a9d70 2024-03-29T10:44:19.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.934+0000 7f52737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:19.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.934+0000 7f52897fa640 4 set_mon_vals no callback set 2024-03-29T10:44:19.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.935+0000 7f52737fe640 1 -- 172.21.15.84:0/840006687 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f527c004980 con 0x7f528c0a9d70 2024-03-29T10:44:19.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.935+0000 7f52737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:19.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.935+0000 7f52737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:19.927 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.935+0000 7f52737fe640 10 monclient: dump: 2024-03-29T10:44:19.927 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:19.927 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:19.927 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:19.927 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:19.927 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:19.927 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:19.927 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:19.927 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:19.927 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:19.928 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:19.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.936+0000 7f52737fe640 1 -- 172.21.15.84:0/840006687 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f527c004ae0 con 0x7f528c0a9d70 2024-03-29T10:44:19.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.937+0000 7f52913cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:19.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.937+0000 7f52913cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:19.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.937+0000 7f52913cd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:44:19.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.937+0000 7f52913cd640 1 -- 172.21.15.84:0/840006687 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f528c0b24a0 con 0x7f528c0a9d70 2024-03-29T10:44:19.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.937+0000 7f52737fe640 1 -- 172.21.15.84:0/840006687 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f527c0849d0 con 0x7f528c0a9d70 2024-03-29T10:44:19.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.937+0000 7f528a7fc640 10 monclient: get_auth_request con 0x7f527806de50 auth_method 0 2024-03-29T10:44:19.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.937+0000 7f52737fe640 10 client.4507.objecter ms_dispatch 0x7f528c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:19.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.937+0000 7f52737fe640 3 client.4507.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:19.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.937+0000 7f52737fe640 3 client.4507.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:19.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.938+0000 7f52737fe640 20 client.4507.objecter dump_active .. 0 homeless 2024-03-29T10:44:19.928 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.938+0000 7f52737fe640 10 client.4507.objecter ms_handle_connect 0x7f527806de50 2024-03-29T10:44:19.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.946+0000 7f52737fe640 1 -- 172.21.15.84:0/840006687 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f527c0502b0 con 0x7f528c0a9d70 2024-03-29T10:44:19.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.946+0000 7f52737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:19.929 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:19.946+0000 7f52737fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:20.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:20.312+0000 7f52913cd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:20.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:20.312+0000 7f52913cd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:20.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:20.312+0000 7f52913cd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:44:20.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:20.312+0000 7f52913cd640 1 -- 172.21.15.84:0/840006687 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f528c13c0c0 con 0x7f528c0a9d70 2024-03-29T10:44:20.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:20.313+0000 7f52737fe640 1 -- 172.21.15.84:0/840006687 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f527c0521e0 con 0x7f528c0a9d70 2024-03-29T10:44:20.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:20.313+0000 7f52737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:20.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:20.313+0000 7f52737fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:20.296 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:44:20.296 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:20.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:20.317+0000 7f52913cd640 10 librados: watch_flush enter 2024-03-29T10:44:20.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:20.317+0000 7f52913cd640 10 librados: watch_flush exit 2024-03-29T10:44:20.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:20.317+0000 7f52913cd640 20 client.4507.objecter shutdown clearing up homeless session... 2024-03-29T10:44:20.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:20.317+0000 7f52913cd640 10 client.4507.objecter successfully canceled tick 2024-03-29T10:44:20.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:20.317+0000 7f52913cd640 1 -- 172.21.15.84:0/840006687 >> v1:172.21.15.86:6816/3337225337 conn(0x7f527806de50 legacy=0x7f5278070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:20.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:20.317+0000 7f52913cd640 10 monclient: shutdown 2024-03-29T10:44:20.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:20.317+0000 7f52913cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:20.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:20.317+0000 7f52913cd640 1 -- 172.21.15.84:0/840006687 >> v1:172.21.15.86:6789/0 conn(0x7f528c0a9d70 legacy=0x7f528c0711a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:20.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:20.317+0000 7f52913cd640 1 -- 172.21.15.84:0/840006687 shutdown_connections 2024-03-29T10:44:20.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:20.317+0000 7f52913cd640 1 -- 172.21.15.84:0/840006687 >> 172.21.15.84:0/840006687 conn(0x7f528c066a00 msgr2=0x7f528c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:20.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:20.317+0000 7f52913cd640 1 -- 172.21.15.84:0/840006687 shutdown_connections 2024-03-29T10:44:20.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:20.317+0000 7f52913cd640 1 -- 172.21.15.84:0/840006687 wait complete. 2024-03-29T10:44:20.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:20.318+0000 7f52913cd640 1 librados: shutdown 2024-03-29T10:44:20.316 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:27.124 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:27.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.283+0000 7f3a519ec640 10 monclient: get_monmap_and_config 2024-03-29T10:44:27.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.283+0000 7f3a519ec640 10 monclient: build_initial_monmap 2024-03-29T10:44:27.266 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.283+0000 7f3a519ec640 10 monclient: monmap: 2024-03-29T10:44:27.266 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:44:27.266 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:27.266 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:44:27.284122+0000 2024-03-29T10:44:27.266 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:44:27.284122+0000 2024-03-29T10:44:27.267 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:27.267 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:27.267 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:44:27.267 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:44:27.267 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:44:27.267 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:27.269 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.286+0000 7f3a519ec640 1 Processor -- start 2024-03-29T10:44:27.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.286+0000 7f3a519ec640 1 -- start start 2024-03-29T10:44:27.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.286+0000 7f3a519ec640 10 monclient: init 2024-03-29T10:44:27.270 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.287+0000 7f3a519ec640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:27.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.287+0000 7f3a519ec640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:44:27.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.287+0000 7f3a519ec640 10 monclient(hunting): picked mon.noname-a con 0x7f3a44006a70 addr v1:172.21.15.84:6789/0 2024-03-29T10:44:27.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.287+0000 7f3a519ec640 10 monclient(hunting): picked mon.noname-c con 0x7f3a4402b100 addr v1:172.21.15.86:6789/0 2024-03-29T10:44:27.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.287+0000 7f3a519ec640 10 monclient(hunting): picked mon.noname-b con 0x7f3a44029c30 addr v1:172.21.15.84:6790/0 2024-03-29T10:44:27.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.287+0000 7f3a519ec640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a4402a270 con 0x7f3a44006a70 2024-03-29T10:44:27.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.287+0000 7f3a519ec640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a4402c440 con 0x7f3a4402b100 2024-03-29T10:44:27.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.287+0000 7f3a519ec640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a440c9440 con 0x7f3a44029c30 2024-03-29T10:44:27.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.287+0000 7f3a519ec640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:27.271 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.287+0000 7f3a519ec640 10 monclient(hunting): authenticate will time out at 1369.550234s 2024-03-29T10:44:27.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.290+0000 7f3a511eb640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f3a44029c30 0x7f3a440d74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:35730/0 (socket says 172.21.15.84:35730) 2024-03-29T10:44:27.273 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.290+0000 7f3a511eb640 1 -- 172.21.15.84:0/2040625384 learned_addr learned my addr 172.21.15.84:0/2040625384 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:44:27.280 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.294+0000 7f3a4b7fe640 1 -- 172.21.15.84:0/2040625384 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3862606789 0 0) 0x7f3a4402a270 con 0x7f3a44006a70 2024-03-29T10:44:27.280 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.294+0000 7f3a4b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:27.280 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.294+0000 7f3a4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:27.280 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.294+0000 7f3a4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:27.280 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.294+0000 7f3a4b7fe640 10 monclient(hunting): my global_id is 4548 2024-03-29T10:44:27.280 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.294+0000 7f3a4b7fe640 1 -- 172.21.15.84:0/2040625384 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a3c003520 con 0x7f3a44006a70 2024-03-29T10:44:27.280 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.295+0000 7f3a4b7fe640 1 -- 172.21.15.84:0/2040625384 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4065731435 0 0) 0x7f3a440c9440 con 0x7f3a44029c30 2024-03-29T10:44:27.280 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.295+0000 7f3a4b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:27.280 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.295+0000 7f3a4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:27.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.295+0000 7f3a4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:27.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.295+0000 7f3a4b7fe640 10 monclient(hunting): my global_id is 4643 2024-03-29T10:44:27.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.295+0000 7f3a4b7fe640 1 -- 172.21.15.84:0/2040625384 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a4402a270 con 0x7f3a44029c30 2024-03-29T10:44:27.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.295+0000 7f3a4b7fe640 1 -- 172.21.15.84:0/2040625384 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3364816361 0 0) 0x7f3a4402c440 con 0x7f3a4402b100 2024-03-29T10:44:27.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.295+0000 7f3a4b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:44:27.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.295+0000 7f3a4b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:27.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.295+0000 7f3a4b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:27.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.295+0000 7f3a4b7fe640 10 monclient(hunting): my global_id is 4510 2024-03-29T10:44:27.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.295+0000 7f3a4b7fe640 1 -- 172.21.15.84:0/2040625384 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a440c9440 con 0x7f3a4402b100 2024-03-29T10:44:27.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.295+0000 7f3a4b7fe640 1 -- 172.21.15.84:0/2040625384 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2799412961 0 0) 0x7f3a3c003520 con 0x7f3a44006a70 2024-03-29T10:44:27.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.295+0000 7f3a4b7fe640 1 -- 172.21.15.84:0/2040625384 >> v1:172.21.15.84:6790/0 conn(0x7f3a44029c30 legacy=0x7f3a440d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:27.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.295+0000 7f3a4b7fe640 1 -- 172.21.15.84:0/2040625384 >> v1:172.21.15.86:6789/0 conn(0x7f3a4402b100 legacy=0x7f3a440cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:27.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.295+0000 7f3a4b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:44:27.281 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.295+0000 7f3a4b7fe640 1 monclient: found mon.noname-a 2024-03-29T10:44:27.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.295+0000 7f3a4b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:27.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.295+0000 7f3a4b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T10:44:27.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.295+0000 7f3a4b7fe640 1 -- 172.21.15.84:0/2040625384 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a440c95d0 con 0x7f3a44006a70 2024-03-29T10:44:27.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.295+0000 7f3a4b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:44:27.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.295+0000 7f3a4b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:44:27.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.295+0000 7f3a4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:27.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.295+0000 7f3a4b7fe640 1 -- 172.21.15.84:0/2040625384 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f3a4c04f730 con 0x7f3a44006a70 2024-03-29T10:44:27.282 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.295+0000 7f3a4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:27.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.295+0000 7f3a4b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:27.283 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.295+0000 7f3a4b7fe640 10 monclient: dump: 2024-03-29T10:44:27.283 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:27.283 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:27.283 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:27.283 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:27.283 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:27.284 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:27.284 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:27.284 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:27.284 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:27.284 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:27.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.295+0000 7f3a519ec640 5 monclient: authenticate success, global_id 4548 2024-03-29T10:44:27.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.295+0000 7f3a519ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:27.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.299+0000 7f3a4b7fe640 1 -- 172.21.15.84:0/2040625384 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a4c075f10 con 0x7f3a44006a70 2024-03-29T10:44:27.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.299+0000 7f3a4b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:27.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.299+0000 7f3a4b7fe640 1 -- 172.21.15.84:0/2040625384 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f3a4c0758b0 con 0x7f3a44006a70 2024-03-29T10:44:27.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.299+0000 7f3a4b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:27.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.299+0000 7f3a4b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:27.284 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.299+0000 7f3a4b7fe640 10 monclient: dump: 2024-03-29T10:44:27.284 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:27.284 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:27.284 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:27.285 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:27.285 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:27.285 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:27.285 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:27.285 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:27.285 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:27.285 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:27.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.299+0000 7f3a519ec640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:27.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.299+0000 7f3a519ec640 4 set_mon_vals no callback set 2024-03-29T10:44:27.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.300+0000 7f3a519ec640 10 monclient: shutdown 2024-03-29T10:44:27.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.300+0000 7f3a519ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:27.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.300+0000 7f3a519ec640 1 -- 172.21.15.84:0/2040625384 >> v1:172.21.15.84:6789/0 conn(0x7f3a44006a70 legacy=0x7f3a440c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:27.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.300+0000 7f3a519ec640 1 -- 172.21.15.84:0/2040625384 shutdown_connections 2024-03-29T10:44:27.285 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.300+0000 7f3a519ec640 1 -- 172.21.15.84:0/2040625384 >> 172.21.15.84:0/2040625384 conn(0x7f3a440058f0 msgr2=0x7f3a44005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:27.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.302+0000 7f3a519ec640 1 -- 172.21.15.84:0/2040625384 shutdown_connections 2024-03-29T10:44:27.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.302+0000 7f3a519ec640 1 -- 172.21.15.84:0/2040625384 wait complete. 2024-03-29T10:44:27.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.303+0000 7f3a519ec640 10 monclient: build_initial_monmap 2024-03-29T10:44:27.286 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.303+0000 7f3a519ec640 10 monclient: monmap: 2024-03-29T10:44:27.286 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:44:27.287 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:27.287 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:44:27.287 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:44:27.287 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:27.287 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:27.287 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:44:27.287 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:44:27.287 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:44:27.287 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:27.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.304+0000 7f3a519ec640 1 librados: starting msgr at 2024-03-29T10:44:27.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.304+0000 7f3a519ec640 1 librados: starting objecter 2024-03-29T10:44:27.287 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.304+0000 7f3a519ec640 1 Processor -- start 2024-03-29T10:44:27.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.304+0000 7f3a519ec640 1 -- start start 2024-03-29T10:44:27.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.304+0000 7f3a519ec640 1 librados: setting wanted keys 2024-03-29T10:44:27.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.304+0000 7f3a519ec640 1 librados: calling monclient init 2024-03-29T10:44:27.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.304+0000 7f3a519ec640 10 monclient: init 2024-03-29T10:44:27.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.305+0000 7f3a519ec640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:27.288 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.305+0000 7f3a519ec640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:44:27.289 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.305+0000 7f3a519ec640 10 monclient(hunting): picked mon.noname-b con 0x7f3a44006a70 addr v1:172.21.15.86:6789/0 2024-03-29T10:44:27.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.307+0000 7f3a519ec640 10 monclient(hunting): picked mon.noname-c con 0x7f3a44029c30 addr v1:172.21.15.84:6790/0 2024-03-29T10:44:27.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.307+0000 7f3a519ec640 10 monclient(hunting): picked mon.noname-a con 0x7f3a44161610 addr v1:172.21.15.84:6789/0 2024-03-29T10:44:27.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.307+0000 7f3a519ec640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a44160a70 con 0x7f3a44161610 2024-03-29T10:44:27.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.307+0000 7f3a519ec640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a440c9980 con 0x7f3a44006a70 2024-03-29T10:44:27.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.307+0000 7f3a519ec640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a440cc930 con 0x7f3a44029c30 2024-03-29T10:44:27.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.307+0000 7f3a519ec640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:27.290 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.307+0000 7f3a519ec640 10 monclient(hunting): authenticate will time out at 1369.570538s 2024-03-29T10:44:27.294 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.311+0000 7f3a509ea640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f3a44006a70 0x7f3a44160f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:51327/0 (socket says 172.21.15.84:51327) 2024-03-29T10:44:27.294 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.311+0000 7f3a509ea640 1 -- 172.21.15.84:0/527148734 learned_addr learned my addr 172.21.15.84:0/527148734 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:44:27.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.312+0000 7f3a497fa640 10 client.?.objecter ms_handle_connect 0x7f3a44006a70 2024-03-29T10:44:27.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.312+0000 7f3a497fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:27.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.312+0000 7f3a497fa640 10 client.?.objecter ms_handle_connect 0x7f3a44029c30 2024-03-29T10:44:27.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.312+0000 7f3a497fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:27.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.312+0000 7f3a497fa640 1 -- 172.21.15.84:0/527148734 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837553656 0 0) 0x7f3a440cc930 con 0x7f3a44029c30 2024-03-29T10:44:27.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.312+0000 7f3a497fa640 10 monclient(hunting): _negotiate 2024-03-29T10:44:27.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.312+0000 7f3a497fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:27.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.312+0000 7f3a497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:27.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.312+0000 7f3a497fa640 10 monclient(hunting): my global_id is 4646 2024-03-29T10:44:27.295 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.312+0000 7f3a497fa640 1 -- 172.21.15.84:0/527148734 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a30004eb0 con 0x7f3a44029c30 2024-03-29T10:44:27.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.312+0000 7f3a497fa640 1 -- 172.21.15.84:0/527148734 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2311647448 0 0) 0x7f3a440c9980 con 0x7f3a44006a70 2024-03-29T10:44:27.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.312+0000 7f3a497fa640 10 monclient(hunting): _negotiate 2024-03-29T10:44:27.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.312+0000 7f3a497fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:27.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.312+0000 7f3a497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:27.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.312+0000 7f3a497fa640 10 monclient(hunting): my global_id is 4513 2024-03-29T10:44:27.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.312+0000 7f3a497fa640 1 -- 172.21.15.84:0/527148734 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a440cc930 con 0x7f3a44006a70 2024-03-29T10:44:27.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 1 -- 172.21.15.84:0/527148734 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 601740980 0 0) 0x7f3a440cc930 con 0x7f3a44006a70 2024-03-29T10:44:27.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 1 -- 172.21.15.84:0/527148734 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a30005490 con 0x7f3a44006a70 2024-03-29T10:44:27.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 10 client.?.objecter ms_handle_connect 0x7f3a44161610 2024-03-29T10:44:27.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:27.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 1 -- 172.21.15.84:0/527148734 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f3a4c04f400 con 0x7f3a44006a70 2024-03-29T10:44:27.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:27.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:27.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 10 monclient(hunting): dump: 2024-03-29T10:44:27.297 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:27.297 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:27.297 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:27.297 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:27.297 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:27.297 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:27.297 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:27.297 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:27.297 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:27.297 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:27.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:27.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 1 -- 172.21.15.84:0/527148734 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2483926036 0 0) 0x7f3a30004eb0 con 0x7f3a44029c30 2024-03-29T10:44:27.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 1 -- 172.21.15.84:0/527148734 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a30005b10 con 0x7f3a44029c30 2024-03-29T10:44:27.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 1 -- 172.21.15.84:0/527148734 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f3a400018b0 con 0x7f3a44029c30 2024-03-29T10:44:27.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:27.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:27.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 10 monclient(hunting): dump: 2024-03-29T10:44:27.298 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:27.298 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:27.298 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:27.298 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:27.298 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:27.298 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:27.299 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:27.299 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:27.299 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:27.299 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:27.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 1 -- 172.21.15.84:0/527148734 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114616204 0 0) 0x7f3a44160a70 con 0x7f3a44161610 2024-03-29T10:44:27.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 10 monclient(hunting): _negotiate 2024-03-29T10:44:27.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:27.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:27.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 10 monclient(hunting): my global_id is 4551 2024-03-29T10:44:27.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 1 -- 172.21.15.84:0/527148734 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a30005f40 con 0x7f3a44161610 2024-03-29T10:44:27.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 1 -- 172.21.15.84:0/527148734 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1717915140 0 0) 0x7f3a30005b10 con 0x7f3a44029c30 2024-03-29T10:44:27.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 1 -- 172.21.15.84:0/527148734 >> v1:172.21.15.86:6789/0 conn(0x7f3a44006a70 legacy=0x7f3a44160f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:27.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 1 -- 172.21.15.84:0/527148734 >> v1:172.21.15.84:6789/0 conn(0x7f3a44161610 legacy=0x7f3a4416c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:27.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 10 monclient: _finish_hunting 0 2024-03-29T10:44:27.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 1 monclient: found mon.c 2024-03-29T10:44:27.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:27.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:44:27.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 1 -- 172.21.15.84:0/527148734 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a440cd340 con 0x7f3a44029c30 2024-03-29T10:44:27.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 10 monclient: _finish_auth 0 2024-03-29T10:44:27.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 10 monclient: _check_auth_tickets 2024-03-29T10:44:27.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:27.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a519ec640 5 monclient: authenticate success, global_id 4646 2024-03-29T10:44:27.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a519ec640 10 monclient: _renew_subs 2024-03-29T10:44:27.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a519ec640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:44:27.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a519ec640 1 -- 172.21.15.84:0/527148734 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a440ca740 con 0x7f3a44029c30 2024-03-29T10:44:27.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a519ec640 10 client.4646.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:27.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a519ec640 10 monclient: _renew_subs 2024-03-29T10:44:27.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a519ec640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:44:27.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a519ec640 1 -- 172.21.15.84:0/527148734 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a44032510 con 0x7f3a44029c30 2024-03-29T10:44:27.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.313+0000 7f3a519ec640 1 librados: init done 2024-03-29T10:44:27.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.315+0000 7f3a497fa640 1 -- 172.21.15.84:0/527148734 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3a40001d40 con 0x7f3a44029c30 2024-03-29T10:44:27.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.315+0000 7f3a497fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:27.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.315+0000 7f3a4affd640 4 set_mon_vals no callback set 2024-03-29T10:44:27.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.315+0000 7f3a519ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:27.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.316+0000 7f3a497fa640 1 -- 172.21.15.84:0/527148734 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f3a400018b0 con 0x7f3a44029c30 2024-03-29T10:44:27.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.316+0000 7f3a519ec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:27.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.316+0000 7f3a519ec640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:44:27.302 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.316+0000 7f3a519ec640 1 -- 172.21.15.84:0/527148734 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a2c005080 con 0x7f3a44029c30 2024-03-29T10:44:27.302 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.316+0000 7f3a497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:27.302 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.316+0000 7f3a497fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:27.302 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.316+0000 7f3a497fa640 10 monclient: dump: 2024-03-29T10:44:27.302 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:27.302 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:27.302 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:27.302 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:27.302 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:27.302 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:27.302 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:27.302 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:27.302 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:27.302 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:27.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.320+0000 7f3a497fa640 1 -- 172.21.15.84:0/527148734 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f3a400194e0 con 0x7f3a44029c30 2024-03-29T10:44:27.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.320+0000 7f3a497fa640 1 -- 172.21.15.84:0/527148734 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f3a40083380 con 0x7f3a44029c30 2024-03-29T10:44:27.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.320+0000 7f3a497fa640 10 client.4646.objecter ms_dispatch 0x7f3a4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:27.303 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.320+0000 7f3a497fa640 3 client.4646.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:27.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.320+0000 7f3a497fa640 3 client.4646.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:27.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.320+0000 7f3a497fa640 20 client.4646.objecter dump_active .. 0 homeless 2024-03-29T10:44:27.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.320+0000 7f3a509ea640 10 monclient: get_auth_request con 0x7f3a3006e910 auth_method 0 2024-03-29T10:44:27.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.321+0000 7f3a497fa640 10 client.4646.objecter ms_handle_connect 0x7f3a3006e910 2024-03-29T10:44:27.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.329+0000 7f3a497fa640 1 -- 172.21.15.84:0/527148734 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3a4004ec60 con 0x7f3a44029c30 2024-03-29T10:44:27.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.329+0000 7f3a497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:27.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.329+0000 7f3a497fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:27.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.665+0000 7f3a519ec640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:27.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.665+0000 7f3a519ec640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:27.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.665+0000 7f3a519ec640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:44:27.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.665+0000 7f3a519ec640 1 -- 172.21.15.84:0/527148734 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a2c005330 con 0x7f3a44029c30 2024-03-29T10:44:27.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.669+0000 7f3a497fa640 1 -- 172.21.15.84:0/527148734 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f3a40050b90 con 0x7f3a44029c30 2024-03-29T10:44:27.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.669+0000 7f3a497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:27.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.669+0000 7f3a497fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:27.652 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:44:27.653 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:27.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.673+0000 7f3a519ec640 10 librados: watch_flush enter 2024-03-29T10:44:27.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.673+0000 7f3a519ec640 10 librados: watch_flush exit 2024-03-29T10:44:27.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.673+0000 7f3a519ec640 20 client.4646.objecter shutdown clearing up homeless session... 2024-03-29T10:44:27.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.673+0000 7f3a519ec640 10 client.4646.objecter successfully canceled tick 2024-03-29T10:44:27.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.675+0000 7f3a519ec640 1 -- 172.21.15.84:0/527148734 >> v1:172.21.15.86:6816/3337225337 conn(0x7f3a3006e910 legacy=0x7f3a30070dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:27.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.675+0000 7f3a519ec640 10 monclient: shutdown 2024-03-29T10:44:27.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.675+0000 7f3a519ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:27.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.675+0000 7f3a519ec640 1 -- 172.21.15.84:0/527148734 >> v1:172.21.15.84:6790/0 conn(0x7f3a44029c30 legacy=0x7f3a44031e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:27.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.675+0000 7f3a519ec640 1 -- 172.21.15.84:0/527148734 shutdown_connections 2024-03-29T10:44:27.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.675+0000 7f3a519ec640 1 -- 172.21.15.84:0/527148734 >> 172.21.15.84:0/527148734 conn(0x7f3a440058f0 msgr2=0x7f3a440372e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:27.661 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.678+0000 7f3a519ec640 1 -- 172.21.15.84:0/527148734 shutdown_connections 2024-03-29T10:44:27.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.679+0000 7f3a519ec640 1 -- 172.21.15.84:0/527148734 wait complete. 2024-03-29T10:44:27.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:27.681+0000 7f3a519ec640 1 librados: shutdown 2024-03-29T10:44:27.681 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:34.488 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:34.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.656+0000 7f7116962640 10 monclient: get_monmap_and_config 2024-03-29T10:44:34.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.656+0000 7f7116962640 10 monclient: build_initial_monmap 2024-03-29T10:44:34.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.656+0000 7f7116962640 10 monclient: monmap: 2024-03-29T10:44:34.640 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:44:34.640 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:34.640 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:44:34.657690+0000 2024-03-29T10:44:34.640 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:44:34.657690+0000 2024-03-29T10:44:34.640 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:34.640 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:34.640 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:44:34.640 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:44:34.640 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:44:34.640 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:34.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.657+0000 7f7116962640 1 Processor -- start 2024-03-29T10:44:34.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.658+0000 7f7116962640 1 -- start start 2024-03-29T10:44:34.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.658+0000 7f7116962640 10 monclient: init 2024-03-29T10:44:34.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.661+0000 7f7116962640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:34.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.661+0000 7f7116962640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:44:34.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.661+0000 7f7116962640 10 monclient(hunting): picked mon.noname-b con 0x7f71100b4820 addr v1:172.21.15.84:6790/0 2024-03-29T10:44:34.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.661+0000 7f7116962640 10 monclient(hunting): picked mon.noname-a con 0x7f71100a9ce0 addr v1:172.21.15.84:6789/0 2024-03-29T10:44:34.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.661+0000 7f7116962640 10 monclient(hunting): picked mon.noname-c con 0x7f71100b3cd0 addr v1:172.21.15.86:6789/0 2024-03-29T10:44:34.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.661+0000 7f7116962640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71100b4180 con 0x7f71100a9ce0 2024-03-29T10:44:34.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.661+0000 7f7116962640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71100aaf60 con 0x7f71100b3cd0 2024-03-29T10:44:34.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.661+0000 7f7116962640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f711013c200 con 0x7f71100b4820 2024-03-29T10:44:34.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.661+0000 7f7116962640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:34.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.661+0000 7f7116962640 10 monclient(hunting): authenticate will time out at 1376.924922s 2024-03-29T10:44:34.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.664+0000 7f7115960640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f71100b4820 0x7f711013baf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:41724/0 (socket says 172.21.15.84:41724) 2024-03-29T10:44:34.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.664+0000 7f7115960640 1 -- 172.21.15.84:0/3690248358 learned_addr learned my addr 172.21.15.84:0/3690248358 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:44:34.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.664+0000 7f711495e640 1 -- 172.21.15.84:0/3690248358 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588048207 0 0) 0x7f711013c200 con 0x7f71100b4820 2024-03-29T10:44:34.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.664+0000 7f711495e640 10 monclient(hunting): _negotiate 2024-03-29T10:44:34.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.664+0000 7f711495e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:34.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.664+0000 7f711495e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:34.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.664+0000 7f711495e640 10 monclient(hunting): my global_id is 4649 2024-03-29T10:44:34.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.664+0000 7f711495e640 1 -- 172.21.15.84:0/3690248358 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70fc003520 con 0x7f71100b4820 2024-03-29T10:44:34.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.668+0000 7f711495e640 1 -- 172.21.15.84:0/3690248358 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4230177205 0 0) 0x7f70fc003520 con 0x7f71100b4820 2024-03-29T10:44:34.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.668+0000 7f711495e640 1 -- 172.21.15.84:0/3690248358 >> v1:172.21.15.86:6789/0 conn(0x7f71100b3cd0 legacy=0x7f711014d7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:44:34.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.668+0000 7f711495e640 1 -- 172.21.15.84:0/3690248358 >> v1:172.21.15.84:6789/0 conn(0x7f71100a9ce0 legacy=0x7f7110145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:44:34.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.668+0000 7f711495e640 10 monclient: _finish_hunting 0 2024-03-29T10:44:34.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.668+0000 7f711495e640 1 monclient: found mon.noname-b 2024-03-29T10:44:34.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.668+0000 7f711495e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:34.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.668+0000 7f711495e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:44:34.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.668+0000 7f711495e640 1 -- 172.21.15.84:0/3690248358 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7110145e80 con 0x7f71100b4820 2024-03-29T10:44:34.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.668+0000 7f711495e640 10 monclient: _finish_auth 0 2024-03-29T10:44:34.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.668+0000 7f711495e640 10 monclient: _check_auth_tickets 2024-03-29T10:44:34.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.668+0000 7f711495e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:34.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.668+0000 7f711495e640 1 -- 172.21.15.84:0/3690248358 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f710c001690 con 0x7f71100b4820 2024-03-29T10:44:34.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.668+0000 7f711495e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:34.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.668+0000 7f711495e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:34.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.668+0000 7f711495e640 10 monclient: dump: 2024-03-29T10:44:34.653 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:34.653 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:34.653 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:34.654 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:34.654 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:34.654 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:34.654 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:34.654 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:34.654 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:34.654 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:34.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.668+0000 7f711495e640 1 -- 172.21.15.84:0/3690248358 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f710c001860 con 0x7f71100b4820 2024-03-29T10:44:34.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.668+0000 7f711495e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:34.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.668+0000 7f711495e640 1 -- 172.21.15.84:0/3690248358 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f710c001cb0 con 0x7f71100b4820 2024-03-29T10:44:34.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.668+0000 7f711495e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:34.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.668+0000 7f711495e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:34.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.668+0000 7f711495e640 10 monclient: dump: 2024-03-29T10:44:34.654 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:34.655 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:34.655 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:34.655 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:34.655 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:34.655 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:34.655 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:34.655 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:34.655 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:34.655 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:34.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.668+0000 7f7116962640 5 monclient: authenticate success, global_id 4649 2024-03-29T10:44:34.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.668+0000 7f7116962640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:34.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.668+0000 7f7116962640 4 set_mon_vals no callback set 2024-03-29T10:44:34.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.669+0000 7f7116962640 10 monclient: shutdown 2024-03-29T10:44:34.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.669+0000 7f7116962640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:34.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.669+0000 7f7116962640 1 -- 172.21.15.84:0/3690248358 >> v1:172.21.15.84:6790/0 conn(0x7f71100b4820 legacy=0x7f711013baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:34.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.670+0000 7f7116962640 1 -- 172.21.15.84:0/3690248358 shutdown_connections 2024-03-29T10:44:34.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.670+0000 7f7116962640 1 -- 172.21.15.84:0/3690248358 >> 172.21.15.84:0/3690248358 conn(0x7f711006d070 msgr2=0x7f7110066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:34.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.671+0000 7f7116962640 1 -- 172.21.15.84:0/3690248358 shutdown_connections 2024-03-29T10:44:34.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.675+0000 7f7116962640 1 -- 172.21.15.84:0/3690248358 wait complete. 2024-03-29T10:44:34.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.676+0000 7f7116962640 10 monclient: build_initial_monmap 2024-03-29T10:44:34.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.676+0000 7f7116962640 10 monclient: monmap: 2024-03-29T10:44:34.660 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:44:34.660 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:34.660 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:44:34.660 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:44:34.660 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:34.660 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:34.661 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:44:34.661 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:44:34.661 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:44:34.661 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:34.661 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.676+0000 7f7116962640 1 librados: starting msgr at 2024-03-29T10:44:34.661 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.676+0000 7f7116962640 1 librados: starting objecter 2024-03-29T10:44:34.661 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.677+0000 7f7116962640 1 Processor -- start 2024-03-29T10:44:34.662 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.679+0000 7f7116962640 1 -- start start 2024-03-29T10:44:34.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.679+0000 7f7116962640 1 librados: setting wanted keys 2024-03-29T10:44:34.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.679+0000 7f7116962640 1 librados: calling monclient init 2024-03-29T10:44:34.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.679+0000 7f7116962640 10 monclient: init 2024-03-29T10:44:34.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.680+0000 7f7116962640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:34.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.680+0000 7f7116962640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:44:34.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.680+0000 7f7116962640 10 monclient(hunting): picked mon.noname-a con 0x7f71100a9ce0 addr v1:172.21.15.84:6789/0 2024-03-29T10:44:34.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.680+0000 7f7116962640 10 monclient(hunting): picked mon.noname-c con 0x7f71100b3cd0 addr v1:172.21.15.84:6790/0 2024-03-29T10:44:34.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.680+0000 7f7116962640 10 monclient(hunting): picked mon.noname-b con 0x7f71100b4820 addr v1:172.21.15.86:6789/0 2024-03-29T10:44:34.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.680+0000 7f7116962640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f711013a680 con 0x7f71100a9ce0 2024-03-29T10:44:34.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.680+0000 7f7116962640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7110144c10 con 0x7f71100b4820 2024-03-29T10:44:34.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.680+0000 7f7116962640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f711013bd70 con 0x7f71100b3cd0 2024-03-29T10:44:34.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.680+0000 7f7116962640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:34.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.680+0000 7f7116962640 10 monclient(hunting): authenticate will time out at 1376.943419s 2024-03-29T10:44:34.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.683+0000 7f711515f640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f71100b3cd0 0x7f7110132bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:56652/0 (socket says 172.21.15.84:56652) 2024-03-29T10:44:34.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.683+0000 7f711515f640 1 -- 172.21.15.84:0/3025665503 learned_addr learned my addr 172.21.15.84:0/3025665503 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:44:34.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.683+0000 7f71067fc640 10 client.?.objecter ms_handle_connect 0x7f71100b3cd0 2024-03-29T10:44:34.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.683+0000 7f71067fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:34.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.683+0000 7f71067fc640 10 client.?.objecter ms_handle_connect 0x7f71100a9ce0 2024-03-29T10:44:34.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.683+0000 7f71067fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:34.670 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.684+0000 7f71067fc640 10 client.?.objecter ms_handle_connect 0x7f71100b4820 2024-03-29T10:44:34.670 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.684+0000 7f71067fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:34.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.684+0000 7f71067fc640 1 -- 172.21.15.84:0/3025665503 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3702543602 0 0) 0x7f711013bd70 con 0x7f71100b3cd0 2024-03-29T10:44:34.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.684+0000 7f71067fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:34.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.684+0000 7f71067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:34.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.684+0000 7f71067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:34.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.684+0000 7f71067fc640 10 monclient(hunting): my global_id is 4652 2024-03-29T10:44:34.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.684+0000 7f71067fc640 1 -- 172.21.15.84:0/3025665503 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70f4004eb0 con 0x7f71100b3cd0 2024-03-29T10:44:34.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.684+0000 7f71067fc640 1 -- 172.21.15.84:0/3025665503 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 146902 0 0) 0x7f711013a680 con 0x7f71100a9ce0 2024-03-29T10:44:34.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.684+0000 7f71067fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:34.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.684+0000 7f71067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:34.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.684+0000 7f71067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:34.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.684+0000 7f71067fc640 10 monclient(hunting): my global_id is 4554 2024-03-29T10:44:34.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.684+0000 7f71067fc640 1 -- 172.21.15.84:0/3025665503 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f711013bd70 con 0x7f71100a9ce0 2024-03-29T10:44:34.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.684+0000 7f71067fc640 1 -- 172.21.15.84:0/3025665503 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1882118048 0 0) 0x7f7110144c10 con 0x7f71100b4820 2024-03-29T10:44:34.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.684+0000 7f71067fc640 10 monclient(hunting): _negotiate 2024-03-29T10:44:34.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.684+0000 7f71067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:34.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.684+0000 7f71067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:34.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.684+0000 7f71067fc640 10 monclient(hunting): my global_id is 4516 2024-03-29T10:44:34.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.684+0000 7f71067fc640 1 -- 172.21.15.84:0/3025665503 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f711013a680 con 0x7f71100b4820 2024-03-29T10:44:34.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.685+0000 7f71067fc640 1 -- 172.21.15.84:0/3025665503 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 382847699 0 0) 0x7f711013a680 con 0x7f71100b4820 2024-03-29T10:44:34.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.685+0000 7f71067fc640 1 -- 172.21.15.84:0/3025665503 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7110144c10 con 0x7f71100b4820 2024-03-29T10:44:34.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.685+0000 7f71067fc640 1 -- 172.21.15.84:0/3025665503 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1953442921 0 0) 0x7f711013bd70 con 0x7f71100a9ce0 2024-03-29T10:44:34.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.685+0000 7f71067fc640 1 -- 172.21.15.84:0/3025665503 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f711013a680 con 0x7f71100a9ce0 2024-03-29T10:44:34.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.686+0000 7f71067fc640 1 -- 172.21.15.84:0/3025665503 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f7108001ab0 con 0x7f71100b4820 2024-03-29T10:44:34.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.686+0000 7f71067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:34.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.686+0000 7f71067fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:34.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.686+0000 7f71067fc640 10 monclient(hunting): dump: 2024-03-29T10:44:34.672 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:34.673 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:34.673 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:34.673 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:34.673 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:34.673 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:34.673 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:34.673 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:34.673 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:34.673 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:34.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.687+0000 7f71067fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:34.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.687+0000 7f71067fc640 1 -- 172.21.15.84:0/3025665503 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f710c001b70 con 0x7f71100a9ce0 2024-03-29T10:44:34.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.687+0000 7f71067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:34.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.687+0000 7f71067fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:34.673 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.687+0000 7f71067fc640 10 monclient(hunting): dump: 2024-03-29T10:44:34.674 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:34.674 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:34.674 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:34.674 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:34.674 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:34.674 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:34.674 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:34.674 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:34.674 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:34.674 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:34.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.687+0000 7f71067fc640 1 -- 172.21.15.84:0/3025665503 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2299937068 0 0) 0x7f7110144c10 con 0x7f71100b4820 2024-03-29T10:44:34.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.687+0000 7f71067fc640 1 -- 172.21.15.84:0/3025665503 >> v1:172.21.15.84:6790/0 conn(0x7f71100b3cd0 legacy=0x7f7110132bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:34.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.687+0000 7f71067fc640 1 -- 172.21.15.84:0/3025665503 >> v1:172.21.15.84:6789/0 conn(0x7f71100a9ce0 legacy=0x7f7110144500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:34.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.687+0000 7f71067fc640 10 monclient: _finish_hunting 0 2024-03-29T10:44:34.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.687+0000 7f71067fc640 1 monclient: found mon.b 2024-03-29T10:44:34.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.687+0000 7f71067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:34.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.687+0000 7f71067fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:44:34.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.687+0000 7f71067fc640 1 -- 172.21.15.84:0/3025665503 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7110134790 con 0x7f71100b4820 2024-03-29T10:44:34.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.687+0000 7f71067fc640 10 monclient: _finish_auth 0 2024-03-29T10:44:34.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.687+0000 7f71067fc640 10 monclient: _check_auth_tickets 2024-03-29T10:44:34.675 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.687+0000 7f71067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:34.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.688+0000 7f71067fc640 1 -- 172.21.15.84:0/3025665503 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7108001c10 con 0x7f71100b4820 2024-03-29T10:44:34.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.688+0000 7f71067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:34.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.688+0000 7f71077fe640 4 set_mon_vals no callback set 2024-03-29T10:44:34.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.689+0000 7f7116962640 5 monclient: authenticate success, global_id 4516 2024-03-29T10:44:34.677 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.689+0000 7f7116962640 10 monclient: _renew_subs 2024-03-29T10:44:34.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.689+0000 7f7116962640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:44:34.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.689+0000 7f7116962640 1 -- 172.21.15.84:0/3025665503 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7110134a00 con 0x7f71100b4820 2024-03-29T10:44:34.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.689+0000 7f7116962640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:34.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.689+0000 7f7116962640 10 monclient: _renew_subs 2024-03-29T10:44:34.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.689+0000 7f7116962640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:44:34.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.689+0000 7f7116962640 1 -- 172.21.15.84:0/3025665503 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f71101334e0 con 0x7f71100b4820 2024-03-29T10:44:34.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.689+0000 7f7116962640 1 librados: init done 2024-03-29T10:44:34.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.691+0000 7f71067fc640 1 -- 172.21.15.84:0/3025665503 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f7108003a10 con 0x7f71100b4820 2024-03-29T10:44:34.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.691+0000 7f71067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:34.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.691+0000 7f71067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:44:34.678 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.691+0000 7f71067fc640 10 monclient: dump: 2024-03-29T10:44:34.678 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:34.678 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:34.678 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:34.679 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:34.679 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:34.679 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:34.679 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:34.679 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:34.679 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:34.679 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:34.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.692+0000 7f7116962640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:34.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.692+0000 7f7116962640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:34.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.692+0000 7f7116962640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:44:34.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.692+0000 7f7116962640 1 -- 172.21.15.84:0/3025665503 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f711013a880 con 0x7f71100b4820 2024-03-29T10:44:34.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.694+0000 7f71067fc640 1 -- 172.21.15.84:0/3025665503 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f71080199e0 con 0x7f71100b4820 2024-03-29T10:44:34.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.694+0000 7f71067fc640 1 -- 172.21.15.84:0/3025665503 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f7108083710 con 0x7f71100b4820 2024-03-29T10:44:34.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.694+0000 7f71067fc640 10 client.4516.objecter ms_dispatch 0x7f7110000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:34.679 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.694+0000 7f71067fc640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:34.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.694+0000 7f71067fc640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:34.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.694+0000 7f71067fc640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-29T10:44:34.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.695+0000 7f7115960640 10 monclient: get_auth_request con 0x7f70f406e100 auth_method 0 2024-03-29T10:44:34.680 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.696+0000 7f71067fc640 10 client.4516.objecter ms_handle_connect 0x7f70f406e100 2024-03-29T10:44:34.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.698+0000 7f71067fc640 1 -- 172.21.15.84:0/3025665503 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f710804ee20 con 0x7f71100b4820 2024-03-29T10:44:34.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.698+0000 7f71067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:34.682 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:34.698+0000 7f71067fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:35.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:35.046+0000 7f7116962640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:35.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:35.046+0000 7f7116962640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:35.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:35.046+0000 7f7116962640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:44:35.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:35.046+0000 7f7116962640 1 -- 172.21.15.84:0/3025665503 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f711013a880 con 0x7f71100b4820 2024-03-29T10:44:35.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:35.047+0000 7f71067fc640 1 -- 172.21.15.84:0/3025665503 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f711013a880 con 0x7f71100b4820 2024-03-29T10:44:35.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:35.047+0000 7f71067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:35.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:35.047+0000 7f71067fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:35.031 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:44:35.031 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:35.034 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:35.052+0000 7f7116962640 10 librados: watch_flush enter 2024-03-29T10:44:35.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:35.052+0000 7f7116962640 10 librados: watch_flush exit 2024-03-29T10:44:35.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:35.052+0000 7f7116962640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-29T10:44:35.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:35.052+0000 7f7116962640 10 client.4516.objecter successfully canceled tick 2024-03-29T10:44:35.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:35.052+0000 7f7116962640 1 -- 172.21.15.84:0/3025665503 >> v1:172.21.15.86:6816/3337225337 conn(0x7f70f406e100 legacy=0x7f70f40705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:35.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:35.052+0000 7f7116962640 10 monclient: shutdown 2024-03-29T10:44:35.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:35.052+0000 7f7116962640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:35.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:35.052+0000 7f7116962640 1 -- 172.21.15.84:0/3025665503 >> v1:172.21.15.86:6789/0 conn(0x7f71100b4820 legacy=0x7f7110143200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:35.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:35.052+0000 7f7116962640 1 -- 172.21.15.84:0/3025665503 shutdown_connections 2024-03-29T10:44:35.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:35.052+0000 7f7116962640 1 -- 172.21.15.84:0/3025665503 >> 172.21.15.84:0/3025665503 conn(0x7f711006d070 msgr2=0x7f71100b8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:35.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:35.058+0000 7f7116962640 1 -- 172.21.15.84:0/3025665503 shutdown_connections 2024-03-29T10:44:35.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:35.058+0000 7f7116962640 1 -- 172.21.15.84:0/3025665503 wait complete. 2024-03-29T10:44:35.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:35.058+0000 7f7116962640 1 librados: shutdown 2024-03-29T10:44:35.059 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:41.866 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:42.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.019+0000 7f98ecd3d640 10 monclient: get_monmap_and_config 2024-03-29T10:44:42.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.019+0000 7f98ecd3d640 10 monclient: build_initial_monmap 2024-03-29T10:44:42.003 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.019+0000 7f98ecd3d640 10 monclient: monmap: 2024-03-29T10:44:42.003 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:44:42.003 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:42.003 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:44:42.021392+0000 2024-03-29T10:44:42.004 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:44:42.021392+0000 2024-03-29T10:44:42.004 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:42.004 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:42.004 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:44:42.004 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:44:42.004 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:44:42.004 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:42.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.020+0000 7f98ecd3d640 1 Processor -- start 2024-03-29T10:44:42.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.020+0000 7f98ecd3d640 1 -- start start 2024-03-29T10:44:42.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.020+0000 7f98ecd3d640 10 monclient: init 2024-03-29T10:44:42.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.021+0000 7f98ecd3d640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:42.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.021+0000 7f98ecd3d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:44:42.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.021+0000 7f98ecd3d640 10 monclient(hunting): picked mon.noname-c con 0x7f98e81422b0 addr v1:172.21.15.86:6789/0 2024-03-29T10:44:42.006 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.021+0000 7f98ecd3d640 10 monclient(hunting): picked mon.noname-b con 0x7f98e8142690 addr v1:172.21.15.84:6790/0 2024-03-29T10:44:42.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.021+0000 7f98ecd3d640 10 monclient(hunting): picked mon.noname-a con 0x7f98e80b17d0 addr v1:172.21.15.84:6789/0 2024-03-29T10:44:42.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.021+0000 7f98ecd3d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e8142c80 con 0x7f98e80b17d0 2024-03-29T10:44:42.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.021+0000 7f98ecd3d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e80aae40 con 0x7f98e81422b0 2024-03-29T10:44:42.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.021+0000 7f98ecd3d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e813fbd0 con 0x7f98e8142690 2024-03-29T10:44:42.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.021+0000 7f98ecd3d640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:42.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.021+0000 7f98ecd3d640 10 monclient(hunting): authenticate will time out at 1384.285609s 2024-03-29T10:44:42.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.021+0000 7f98e5d9b640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f98e8142690 0x7f98e80b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:49592/0 (socket says 172.21.15.84:49592) 2024-03-29T10:44:42.007 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.021+0000 7f98e5d9b640 1 -- 172.21.15.84:0/129651063 learned_addr learned my addr 172.21.15.84:0/129651063 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:44:42.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.022+0000 7f98e559a640 1 -- 172.21.15.84:0/129651063 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607188434 0 0) 0x7f98e813fbd0 con 0x7f98e8142690 2024-03-29T10:44:42.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.022+0000 7f98e559a640 10 monclient(hunting): _negotiate 2024-03-29T10:44:42.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.022+0000 7f98e559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:42.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.022+0000 7f98e559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:42.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.022+0000 7f98e559a640 10 monclient(hunting): my global_id is 4655 2024-03-29T10:44:42.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.022+0000 7f98e559a640 1 -- 172.21.15.84:0/129651063 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98d0003520 con 0x7f98e8142690 2024-03-29T10:44:42.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.022+0000 7f98e559a640 1 -- 172.21.15.84:0/129651063 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1962474074 0 0) 0x7f98d0003520 con 0x7f98e8142690 2024-03-29T10:44:42.009 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.022+0000 7f98e559a640 1 -- 172.21.15.84:0/129651063 >> v1:172.21.15.86:6789/0 conn(0x7f98e81422b0 legacy=0x7f98e813f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:44:42.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.024+0000 7f98e559a640 1 -- 172.21.15.84:0/129651063 >> v1:172.21.15.84:6789/0 conn(0x7f98e80b17d0 legacy=0x7f98e80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:42.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.024+0000 7f98e559a640 10 monclient: _finish_hunting 0 2024-03-29T10:44:42.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.024+0000 7f98e559a640 1 monclient: found mon.noname-b 2024-03-29T10:44:42.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.024+0000 7f98e559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:42.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.024+0000 7f98e559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:44:42.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.024+0000 7f98e559a640 1 -- 172.21.15.84:0/129651063 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e8151940 con 0x7f98e8142690 2024-03-29T10:44:42.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.024+0000 7f98e559a640 10 monclient: _finish_auth 0 2024-03-29T10:44:42.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.024+0000 7f98e559a640 10 monclient: _check_auth_tickets 2024-03-29T10:44:42.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.024+0000 7f98e559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:42.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.024+0000 7f98e559a640 1 -- 172.21.15.84:0/129651063 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f98d8001a10 con 0x7f98e8142690 2024-03-29T10:44:42.010 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.024+0000 7f98e559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:42.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.024+0000 7f98e559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:44:42.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.024+0000 7f98e559a640 10 monclient: dump: 2024-03-29T10:44:42.011 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:42.011 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:42.011 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:42.011 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:42.011 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:42.011 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:42.011 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:42.011 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:42.011 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:42.011 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:42.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.024+0000 7f98ecd3d640 5 monclient: authenticate success, global_id 4655 2024-03-29T10:44:42.011 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.024+0000 7f98ecd3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:42.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.024+0000 7f98e559a640 1 -- 172.21.15.84:0/129651063 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98d8001b70 con 0x7f98e8142690 2024-03-29T10:44:42.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.024+0000 7f98e559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:42.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.024+0000 7f98e559a640 1 -- 172.21.15.84:0/129651063 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f98d8001f80 con 0x7f98e8142690 2024-03-29T10:44:42.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.024+0000 7f98e559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:42.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.024+0000 7f98e559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:42.012 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.025+0000 7f98e559a640 10 monclient: dump: 2024-03-29T10:44:42.012 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:42.012 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:42.012 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:42.012 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:42.012 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:42.012 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:42.012 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:42.012 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:42.013 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:42.013 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:42.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.026+0000 7f98ecd3d640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:42.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.026+0000 7f98ecd3d640 4 set_mon_vals no callback set 2024-03-29T10:44:42.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.027+0000 7f98ecd3d640 10 monclient: shutdown 2024-03-29T10:44:42.014 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.027+0000 7f98ecd3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:42.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.027+0000 7f98ecd3d640 1 -- 172.21.15.84:0/129651063 >> v1:172.21.15.84:6790/0 conn(0x7f98e8142690 legacy=0x7f98e80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:42.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.027+0000 7f98ecd3d640 1 -- 172.21.15.84:0/129651063 shutdown_connections 2024-03-29T10:44:42.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.027+0000 7f98ecd3d640 1 -- 172.21.15.84:0/129651063 >> 172.21.15.84:0/129651063 conn(0x7f98e8066a00 msgr2=0x7f98e8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:42.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.027+0000 7f98ecd3d640 1 -- 172.21.15.84:0/129651063 shutdown_connections 2024-03-29T10:44:42.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.027+0000 7f98ecd3d640 1 -- 172.21.15.84:0/129651063 wait complete. 2024-03-29T10:44:42.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.029+0000 7f98ecd3d640 10 monclient: build_initial_monmap 2024-03-29T10:44:42.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.029+0000 7f98ecd3d640 10 monclient: monmap: 2024-03-29T10:44:42.015 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:44:42.015 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:42.015 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:44:42.015 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:44:42.015 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:42.015 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:42.015 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:44:42.016 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:44:42.016 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:44:42.016 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:42.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.029+0000 7f98ecd3d640 1 librados: starting msgr at 2024-03-29T10:44:42.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.029+0000 7f98ecd3d640 1 librados: starting objecter 2024-03-29T10:44:42.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.029+0000 7f98ecd3d640 1 Processor -- start 2024-03-29T10:44:42.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.029+0000 7f98ecd3d640 1 -- start start 2024-03-29T10:44:42.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.029+0000 7f98ecd3d640 1 librados: setting wanted keys 2024-03-29T10:44:42.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.029+0000 7f98ecd3d640 1 librados: calling monclient init 2024-03-29T10:44:42.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.029+0000 7f98ecd3d640 10 monclient: init 2024-03-29T10:44:42.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.030+0000 7f98ecd3d640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:42.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.030+0000 7f98ecd3d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:44:42.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.030+0000 7f98ecd3d640 10 monclient(hunting): picked mon.noname-c con 0x7f98e80b17d0 addr v1:172.21.15.84:6790/0 2024-03-29T10:44:42.016 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.030+0000 7f98ecd3d640 10 monclient(hunting): picked mon.noname-a con 0x7f98e81422b0 addr v1:172.21.15.84:6789/0 2024-03-29T10:44:42.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.030+0000 7f98ecd3d640 10 monclient(hunting): picked mon.noname-b con 0x7f98e8142690 addr v1:172.21.15.86:6789/0 2024-03-29T10:44:42.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.030+0000 7f98ecd3d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e812e9f0 con 0x7f98e81422b0 2024-03-29T10:44:42.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.030+0000 7f98ecd3d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e81309e0 con 0x7f98e8142690 2024-03-29T10:44:42.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.030+0000 7f98ecd3d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e806e860 con 0x7f98e80b17d0 2024-03-29T10:44:42.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.030+0000 7f98ecd3d640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:42.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.030+0000 7f98ecd3d640 10 monclient(hunting): authenticate will time out at 1384.294476s 2024-03-29T10:44:42.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.031+0000 7f98e659c640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f98e80b17d0 0x7f98e8071600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:50310/0 (socket says 172.21.15.84:50310) 2024-03-29T10:44:42.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.031+0000 7f98e659c640 1 -- 172.21.15.84:0/3548865000 learned_addr learned my addr 172.21.15.84:0/3548865000 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:44:42.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.031+0000 7f98d6ffd640 10 client.?.objecter ms_handle_connect 0x7f98e80b17d0 2024-03-29T10:44:42.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.031+0000 7f98d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:42.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.031+0000 7f98d6ffd640 10 client.?.objecter ms_handle_connect 0x7f98e81422b0 2024-03-29T10:44:42.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.031+0000 7f98d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:42.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 1 -- 172.21.15.84:0/3548865000 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968841594 0 0) 0x7f98e806e860 con 0x7f98e80b17d0 2024-03-29T10:44:42.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:42.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:42.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:42.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 10 monclient(hunting): my global_id is 4658 2024-03-29T10:44:42.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 1 -- 172.21.15.84:0/3548865000 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98c8004eb0 con 0x7f98e80b17d0 2024-03-29T10:44:42.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 1 -- 172.21.15.84:0/3548865000 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2723758787 0 0) 0x7f98e812e9f0 con 0x7f98e81422b0 2024-03-29T10:44:42.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:44:42.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:42.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:42.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 10 monclient(hunting): my global_id is 4560 2024-03-29T10:44:42.023 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 1 -- 172.21.15.84:0/3548865000 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98e806e860 con 0x7f98e81422b0 2024-03-29T10:44:42.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 1 -- 172.21.15.84:0/3548865000 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2238110707 0 0) 0x7f98c8004eb0 con 0x7f98e80b17d0 2024-03-29T10:44:42.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 1 -- 172.21.15.84:0/3548865000 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98e812e9f0 con 0x7f98e80b17d0 2024-03-29T10:44:42.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 1 -- 172.21.15.84:0/3548865000 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f98e0001870 con 0x7f98e80b17d0 2024-03-29T10:44:42.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:42.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:42.024 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 10 monclient(hunting): dump: 2024-03-29T10:44:42.024 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:42.024 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:42.024 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:42.024 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:42.024 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:42.024 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:42.024 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:42.025 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:42.025 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:42.025 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:42.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:42.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 1 -- 172.21.15.84:0/3548865000 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1985139334 0 0) 0x7f98e806e860 con 0x7f98e81422b0 2024-03-29T10:44:42.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 1 -- 172.21.15.84:0/3548865000 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98c8004eb0 con 0x7f98e81422b0 2024-03-29T10:44:42.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 1 -- 172.21.15.84:0/3548865000 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f98d8001f40 con 0x7f98e81422b0 2024-03-29T10:44:42.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:42.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:42.025 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 10 monclient(hunting): dump: 2024-03-29T10:44:42.025 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:42.025 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:42.025 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:42.025 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:42.026 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:42.026 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:42.026 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:42.026 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:42.026 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:42.026 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:42.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 1 -- 172.21.15.84:0/3548865000 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 688082480 0 0) 0x7f98e812e9f0 con 0x7f98e80b17d0 2024-03-29T10:44:42.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 1 -- 172.21.15.84:0/3548865000 >> v1:172.21.15.86:6789/0 conn(0x7f98e8142690 legacy=0x7f98e8131580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:42.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 1 -- 172.21.15.84:0/3548865000 >> v1:172.21.15.84:6789/0 conn(0x7f98e81422b0 legacy=0x7f98e8130e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:42.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:44:42.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 1 monclient: found mon.c 2024-03-29T10:44:42.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:42.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:44:42.026 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 1 -- 172.21.15.84:0/3548865000 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e806eb40 con 0x7f98e80b17d0 2024-03-29T10:44:42.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:44:42.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:44:42.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.032+0000 7f98d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:42.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.033+0000 7f98ecd3d640 5 monclient: authenticate success, global_id 4658 2024-03-29T10:44:42.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.033+0000 7f98ecd3d640 10 monclient: _renew_subs 2024-03-29T10:44:42.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.033+0000 7f98ecd3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:44:42.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.033+0000 7f98ecd3d640 1 -- 172.21.15.84:0/3548865000 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98e806ee20 con 0x7f98e80b17d0 2024-03-29T10:44:42.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.033+0000 7f98ecd3d640 10 client.4658.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:42.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.033+0000 7f98ecd3d640 10 monclient: _renew_subs 2024-03-29T10:44:42.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.033+0000 7f98ecd3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:44:42.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.033+0000 7f98ecd3d640 1 -- 172.21.15.84:0/3548865000 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e80b7570 con 0x7f98e80b17d0 2024-03-29T10:44:42.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.033+0000 7f98ecd3d640 1 librados: init done 2024-03-29T10:44:42.027 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.034+0000 7f98ecd3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:42.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.034+0000 7f98ecd3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:42.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.034+0000 7f98ecd3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:44:42.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.034+0000 7f98ecd3d640 1 -- 172.21.15.84:0/3548865000 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e8075f10 con 0x7f98e80b17d0 2024-03-29T10:44:42.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.035+0000 7f98d6ffd640 1 -- 172.21.15.84:0/3548865000 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98e0001d80 con 0x7f98e80b17d0 2024-03-29T10:44:42.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.035+0000 7f98d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:42.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.035+0000 7f98d6ffd640 1 -- 172.21.15.84:0/3548865000 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f98e0004aa0 con 0x7f98e80b17d0 2024-03-29T10:44:42.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.035+0000 7f98d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:42.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.035+0000 7f98d6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:42.028 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.035+0000 7f98d6ffd640 10 monclient: dump: 2024-03-29T10:44:42.028 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:42.028 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:42.028 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:42.028 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:42.029 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:42.029 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:42.029 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:42.029 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:42.029 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:42.029 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:42.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.035+0000 7f98d7fff640 4 set_mon_vals no callback set 2024-03-29T10:44:42.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.036+0000 7f98d6ffd640 1 -- 172.21.15.84:0/3548865000 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f98e0004c40 con 0x7f98e80b17d0 2024-03-29T10:44:42.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.036+0000 7f98d6ffd640 1 -- 172.21.15.84:0/3548865000 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f98e0084a50 con 0x7f98e80b17d0 2024-03-29T10:44:42.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.036+0000 7f98e5d9b640 10 monclient: get_auth_request con 0x7f98c806ddf0 auth_method 0 2024-03-29T10:44:42.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.037+0000 7f98d6ffd640 10 client.4658.objecter ms_dispatch 0x7f98e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:42.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.037+0000 7f98d6ffd640 3 client.4658.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:42.029 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.037+0000 7f98d6ffd640 3 client.4658.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:42.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.037+0000 7f98d6ffd640 20 client.4658.objecter dump_active .. 0 homeless 2024-03-29T10:44:42.030 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.037+0000 7f98d6ffd640 10 client.4658.objecter ms_handle_connect 0x7f98c806ddf0 2024-03-29T10:44:42.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.042+0000 7f98d6ffd640 1 -- 172.21.15.84:0/3548865000 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f98e0050330 con 0x7f98e80b17d0 2024-03-29T10:44:42.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.042+0000 7f98d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:42.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.042+0000 7f98d6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:42.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.346+0000 7f98ecd3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:42.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.346+0000 7f98ecd3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:42.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.346+0000 7f98ecd3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:44:42.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.346+0000 7f98ecd3d640 1 -- 172.21.15.84:0/3548865000 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98e80b2a30 con 0x7f98e80b17d0 2024-03-29T10:44:42.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.347+0000 7f98d6ffd640 1 -- 172.21.15.84:0/3548865000 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f98e0052260 con 0x7f98e80b17d0 2024-03-29T10:44:42.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.347+0000 7f98d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:42.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.347+0000 7f98d6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:42.331 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:44:42.331 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:42.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.350+0000 7f98ecd3d640 10 librados: watch_flush enter 2024-03-29T10:44:42.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.350+0000 7f98ecd3d640 10 librados: watch_flush exit 2024-03-29T10:44:42.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.350+0000 7f98ecd3d640 20 client.4658.objecter shutdown clearing up homeless session... 2024-03-29T10:44:42.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.350+0000 7f98ecd3d640 10 client.4658.objecter successfully canceled tick 2024-03-29T10:44:42.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.350+0000 7f98ecd3d640 1 -- 172.21.15.84:0/3548865000 >> v1:172.21.15.86:6816/3337225337 conn(0x7f98c806ddf0 legacy=0x7f98c80702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:42.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.350+0000 7f98ecd3d640 10 monclient: shutdown 2024-03-29T10:44:42.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.350+0000 7f98ecd3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:42.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.350+0000 7f98ecd3d640 1 -- 172.21.15.84:0/3548865000 >> v1:172.21.15.84:6790/0 conn(0x7f98e80b17d0 legacy=0x7f98e8071600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:42.334 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.351+0000 7f98ecd3d640 1 -- 172.21.15.84:0/3548865000 shutdown_connections 2024-03-29T10:44:42.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.351+0000 7f98ecd3d640 1 -- 172.21.15.84:0/3548865000 >> 172.21.15.84:0/3548865000 conn(0x7f98e8066a00 msgr2=0x7f98e812fa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:42.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.351+0000 7f98ecd3d640 1 -- 172.21.15.84:0/3548865000 shutdown_connections 2024-03-29T10:44:42.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.351+0000 7f98ecd3d640 1 -- 172.21.15.84:0/3548865000 wait complete. 2024-03-29T10:44:42.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:42.351+0000 7f98ecd3d640 1 librados: shutdown 2024-03-29T10:44:42.348 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:49.155 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:49.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.313+0000 7f6e721a4640 10 monclient: get_monmap_and_config 2024-03-29T10:44:49.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.313+0000 7f6e721a4640 10 monclient: build_initial_monmap 2024-03-29T10:44:49.296 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.313+0000 7f6e721a4640 10 monclient: monmap: 2024-03-29T10:44:49.296 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:44:49.296 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:49.296 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:44:49.314060+0000 2024-03-29T10:44:49.296 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:44:49.314060+0000 2024-03-29T10:44:49.296 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:49.296 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:49.296 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:44:49.296 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:44:49.296 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:44:49.296 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:49.297 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.314+0000 7f6e721a4640 1 Processor -- start 2024-03-29T10:44:49.298 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.314+0000 7f6e721a4640 1 -- start start 2024-03-29T10:44:49.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.314+0000 7f6e721a4640 10 monclient: init 2024-03-29T10:44:49.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.315+0000 7f6e721a4640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:49.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.315+0000 7f6e721a4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:44:49.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.316+0000 7f6e721a4640 10 monclient(hunting): picked mon.noname-c con 0x7f6e64006d80 addr v1:172.21.15.86:6789/0 2024-03-29T10:44:49.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.316+0000 7f6e711a2640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f6e64006d80 0x7f6e640cbc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:45138/0 (socket says 172.21.15.84:45138) 2024-03-29T10:44:49.299 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.316+0000 7f6e711a2640 1 -- 172.21.15.84:0/105765163 learned_addr learned my addr 172.21.15.84:0/105765163 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:44:49.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.316+0000 7f6e721a4640 10 monclient(hunting): picked mon.noname-b con 0x7f6e640cc310 addr v1:172.21.15.84:6790/0 2024-03-29T10:44:49.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.316+0000 7f6e721a4640 10 monclient(hunting): picked mon.noname-a con 0x7f6e640c9430 addr v1:172.21.15.84:6789/0 2024-03-29T10:44:49.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.316+0000 7f6e721a4640 1 -- 172.21.15.84:0/105765163 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e64008470 con 0x7f6e640c9430 2024-03-29T10:44:49.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.317+0000 7f6e721a4640 1 -- 172.21.15.84:0/105765163 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e6402be30 con 0x7f6e64006d80 2024-03-29T10:44:49.300 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.317+0000 7f6e721a4640 1 -- 172.21.15.84:0/105765163 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e6402c110 con 0x7f6e640cc310 2024-03-29T10:44:49.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.317+0000 7f6e721a4640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:49.301 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.317+0000 7f6e721a4640 10 monclient(hunting): authenticate will time out at 1391.581107s 2024-03-29T10:44:49.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.320+0000 7f6e63fff640 1 -- 172.21.15.84:0/105765163 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3227892446 0 0) 0x7f6e64008470 con 0x7f6e640c9430 2024-03-29T10:44:49.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.321+0000 7f6e63fff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:49.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.321+0000 7f6e63fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:49.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.321+0000 7f6e63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:49.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.321+0000 7f6e63fff640 10 monclient(hunting): my global_id is 4563 2024-03-29T10:44:49.304 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.321+0000 7f6e63fff640 1 -- 172.21.15.84:0/105765163 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e58003520 con 0x7f6e640c9430 2024-03-29T10:44:49.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.322+0000 7f6e63fff640 1 -- 172.21.15.84:0/105765163 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4079795306 0 0) 0x7f6e58003520 con 0x7f6e640c9430 2024-03-29T10:44:49.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.322+0000 7f6e63fff640 1 -- 172.21.15.84:0/105765163 >> v1:172.21.15.84:6790/0 conn(0x7f6e640cc310 legacy=0x7f6e640c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:44:49.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.322+0000 7f6e63fff640 1 -- 172.21.15.84:0/105765163 >> v1:172.21.15.86:6789/0 conn(0x7f6e64006d80 legacy=0x7f6e640cbc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:49.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.322+0000 7f6e63fff640 10 monclient: _finish_hunting 0 2024-03-29T10:44:49.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.322+0000 7f6e63fff640 1 monclient: found mon.noname-a 2024-03-29T10:44:49.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.322+0000 7f6e63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:49.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.322+0000 7f6e63fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T10:44:49.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.322+0000 7f6e63fff640 1 -- 172.21.15.84:0/105765163 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6402c3f0 con 0x7f6e640c9430 2024-03-29T10:44:49.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.322+0000 7f6e63fff640 10 monclient: _finish_auth 0 2024-03-29T10:44:49.308 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.322+0000 7f6e63fff640 10 monclient: _check_auth_tickets 2024-03-29T10:44:49.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.322+0000 7f6e63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:49.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.322+0000 7f6e63fff640 1 -- 172.21.15.84:0/105765163 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f6e5c0016e0 con 0x7f6e640c9430 2024-03-29T10:44:49.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.322+0000 7f6e63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:49.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.322+0000 7f6e63fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:49.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.322+0000 7f6e63fff640 10 monclient: dump: 2024-03-29T10:44:49.309 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:49.309 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:49.309 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:49.309 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:49.309 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:49.309 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:49.309 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:49.309 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:49.309 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:49.309 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:49.309 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.322+0000 7f6e721a4640 5 monclient: authenticate success, global_id 4563 2024-03-29T10:44:49.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.322+0000 7f6e721a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:49.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.323+0000 7f6e63fff640 1 -- 172.21.15.84:0/105765163 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e5c001840 con 0x7f6e640c9430 2024-03-29T10:44:49.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.323+0000 7f6e63fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:49.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.323+0000 7f6e721a4640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:49.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.323+0000 7f6e721a4640 4 set_mon_vals no callback set 2024-03-29T10:44:49.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.323+0000 7f6e63fff640 1 -- 172.21.15.84:0/105765163 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f6e5c001c90 con 0x7f6e640c9430 2024-03-29T10:44:49.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.323+0000 7f6e63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:49.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.323+0000 7f6e63fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:49.310 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.323+0000 7f6e63fff640 10 monclient: dump: 2024-03-29T10:44:49.310 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:49.310 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:49.310 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:49.310 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:49.310 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:49.310 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:49.311 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:49.311 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:49.311 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:49.311 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:49.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.324+0000 7f6e721a4640 10 monclient: shutdown 2024-03-29T10:44:49.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.324+0000 7f6e721a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:49.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.324+0000 7f6e721a4640 1 -- 172.21.15.84:0/105765163 >> v1:172.21.15.84:6789/0 conn(0x7f6e640c9430 legacy=0x7f6e640d74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:49.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.324+0000 7f6e721a4640 1 -- 172.21.15.84:0/105765163 shutdown_connections 2024-03-29T10:44:49.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.324+0000 7f6e721a4640 1 -- 172.21.15.84:0/105765163 >> 172.21.15.84:0/105765163 conn(0x7f6e64005c00 msgr2=0x7f6e64006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:49.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.324+0000 7f6e721a4640 1 -- 172.21.15.84:0/105765163 shutdown_connections 2024-03-29T10:44:49.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.326+0000 7f6e721a4640 1 -- 172.21.15.84:0/105765163 wait complete. 2024-03-29T10:44:49.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.327+0000 7f6e721a4640 10 monclient: build_initial_monmap 2024-03-29T10:44:49.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.327+0000 7f6e721a4640 10 monclient: monmap: 2024-03-29T10:44:49.312 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:44:49.312 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:49.312 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:44:49.313 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:44:49.313 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:49.313 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:49.313 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:44:49.313 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:44:49.313 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:44:49.313 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:49.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.327+0000 7f6e721a4640 1 librados: starting msgr at 2024-03-29T10:44:49.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.327+0000 7f6e721a4640 1 librados: starting objecter 2024-03-29T10:44:49.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.328+0000 7f6e721a4640 1 Processor -- start 2024-03-29T10:44:49.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.328+0000 7f6e721a4640 1 -- start start 2024-03-29T10:44:49.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.328+0000 7f6e721a4640 1 librados: setting wanted keys 2024-03-29T10:44:49.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.328+0000 7f6e721a4640 1 librados: calling monclient init 2024-03-29T10:44:49.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.328+0000 7f6e721a4640 10 monclient: init 2024-03-29T10:44:49.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.329+0000 7f6e721a4640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:49.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.329+0000 7f6e721a4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:44:49.314 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.329+0000 7f6e721a4640 10 monclient(hunting): picked mon.noname-c con 0x7f6e64006d80 addr v1:172.21.15.84:6790/0 2024-03-29T10:44:49.315 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.330+0000 7f6e721a4640 10 monclient(hunting): picked mon.noname-a con 0x7f6e640c9430 addr v1:172.21.15.84:6789/0 2024-03-29T10:44:49.315 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.330+0000 7f6e721a4640 10 monclient(hunting): picked mon.noname-b con 0x7f6e640cc310 addr v1:172.21.15.86:6789/0 2024-03-29T10:44:49.315 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.330+0000 7f6e721a4640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e640c5260 con 0x7f6e640c9430 2024-03-29T10:44:49.315 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.330+0000 7f6e721a4640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e6402c8b0 con 0x7f6e640cc310 2024-03-29T10:44:49.315 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.330+0000 7f6e721a4640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e6402d900 con 0x7f6e64006d80 2024-03-29T10:44:49.315 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.330+0000 7f6e721a4640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:49.315 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.330+0000 7f6e721a4640 10 monclient(hunting): authenticate will time out at 1391.593324s 2024-03-29T10:44:49.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.333+0000 7f6e711a2640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f6e64006d80 0x7f6e6402cd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:57975/0 (socket says 172.21.15.84:57975) 2024-03-29T10:44:49.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.333+0000 7f6e711a2640 1 -- 172.21.15.84:0/1046148165 learned_addr learned my addr 172.21.15.84:0/1046148165 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:44:49.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.333+0000 7f6e61ffb640 10 client.?.objecter ms_handle_connect 0x7f6e64006d80 2024-03-29T10:44:49.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.333+0000 7f6e61ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:49.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.333+0000 7f6e61ffb640 1 -- 172.21.15.84:0/1046148165 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2464724621 0 0) 0x7f6e6402d900 con 0x7f6e64006d80 2024-03-29T10:44:49.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.333+0000 7f6e61ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:44:49.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.333+0000 7f6e61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:49.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.333+0000 7f6e61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:49.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.333+0000 7f6e61ffb640 10 monclient(hunting): my global_id is 4661 2024-03-29T10:44:49.316 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.333+0000 7f6e61ffb640 1 -- 172.21.15.84:0/1046148165 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e50004eb0 con 0x7f6e64006d80 2024-03-29T10:44:49.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.333+0000 7f6e61ffb640 1 -- 172.21.15.84:0/1046148165 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2345664378 0 0) 0x7f6e50004eb0 con 0x7f6e64006d80 2024-03-29T10:44:49.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.334+0000 7f6e61ffb640 1 -- 172.21.15.84:0/1046148165 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e6402d900 con 0x7f6e64006d80 2024-03-29T10:44:49.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.334+0000 7f6e61ffb640 1 -- 172.21.15.84:0/1046148165 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f6e6c04fdd0 con 0x7f6e64006d80 2024-03-29T10:44:49.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.334+0000 7f6e61ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:49.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.334+0000 7f6e61ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:44:49.317 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.334+0000 7f6e61ffb640 10 monclient(hunting): dump: 2024-03-29T10:44:49.317 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:49.318 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:49.318 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:49.318 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:49.318 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:49.318 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:49.318 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:49.318 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:49.318 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:49.318 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:49.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.334+0000 7f6e61ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:49.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.334+0000 7f6e61ffb640 1 -- 172.21.15.84:0/1046148165 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 909478094 0 0) 0x7f6e6402d900 con 0x7f6e64006d80 2024-03-29T10:44:49.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.334+0000 7f6e61ffb640 1 -- 172.21.15.84:0/1046148165 >> v1:172.21.15.86:6789/0 conn(0x7f6e640cc310 legacy=0x7f6e640c4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:49.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.334+0000 7f6e61ffb640 1 -- 172.21.15.84:0/1046148165 >> v1:172.21.15.84:6789/0 conn(0x7f6e640c9430 legacy=0x7f6e640c4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:49.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.334+0000 7f6e61ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:44:49.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.334+0000 7f6e61ffb640 1 monclient: found mon.c 2024-03-29T10:44:49.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.334+0000 7f6e61ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:49.320 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.334+0000 7f6e61ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:44:49.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.334+0000 7f6e61ffb640 1 -- 172.21.15.84:0/1046148165 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6402da90 con 0x7f6e64006d80 2024-03-29T10:44:49.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.334+0000 7f6e61ffb640 10 monclient: _finish_auth 0 2024-03-29T10:44:49.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.334+0000 7f6e61ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:44:49.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.334+0000 7f6e61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:49.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.337+0000 7f6e721a4640 5 monclient: authenticate success, global_id 4661 2024-03-29T10:44:49.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.337+0000 7f6e721a4640 10 monclient: _renew_subs 2024-03-29T10:44:49.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.337+0000 7f6e721a4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:44:49.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.337+0000 7f6e721a4640 1 -- 172.21.15.84:0/1046148165 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e641640f0 con 0x7f6e64006d80 2024-03-29T10:44:49.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.337+0000 7f6e721a4640 10 client.4661.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:49.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.337+0000 7f6e721a4640 10 monclient: _renew_subs 2024-03-29T10:44:49.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.337+0000 7f6e721a4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:44:49.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.337+0000 7f6e721a4640 1 -- 172.21.15.84:0/1046148165 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e64034960 con 0x7f6e64006d80 2024-03-29T10:44:49.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.337+0000 7f6e721a4640 1 librados: init done 2024-03-29T10:44:49.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.338+0000 7f6e61ffb640 1 -- 172.21.15.84:0/1046148165 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e6c04f400 con 0x7f6e64006d80 2024-03-29T10:44:49.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.338+0000 7f6e61ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:49.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.338+0000 7f6e61ffb640 1 -- 172.21.15.84:0/1046148165 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f6e6c075710 con 0x7f6e64006d80 2024-03-29T10:44:49.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.338+0000 7f6e61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:49.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.338+0000 7f6e61ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:44:49.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.338+0000 7f6e61ffb640 10 monclient: dump: 2024-03-29T10:44:49.323 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:49.323 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:49.323 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:49.323 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:49.323 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:49.323 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:49.323 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:49.323 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:49.323 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:49.323 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:49.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.338+0000 7f6e62ffd640 4 set_mon_vals no callback set 2024-03-29T10:44:49.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.339+0000 7f6e721a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:49.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.339+0000 7f6e721a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:49.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.339+0000 7f6e721a4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:44:49.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.339+0000 7f6e721a4640 1 -- 172.21.15.84:0/1046148165 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e4c005080 con 0x7f6e64006d80 2024-03-29T10:44:49.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.339+0000 7f6e61ffb640 1 -- 172.21.15.84:0/1046148165 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f6e6c067bf0 con 0x7f6e64006d80 2024-03-29T10:44:49.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.340+0000 7f6e61ffb640 1 -- 172.21.15.84:0/1046148165 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 4110261992 0 0) 0x7f6e6c0ff6c0 con 0x7f6e64006d80 2024-03-29T10:44:49.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.340+0000 7f6e61ffb640 10 client.4661.objecter ms_dispatch 0x7f6e6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:44:49.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.340+0000 7f6e61ffb640 3 client.4661.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:44:49.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.340+0000 7f6e61ffb640 3 client.4661.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:44:49.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.340+0000 7f6e61ffb640 20 client.4661.objecter dump_active .. 0 homeless 2024-03-29T10:44:49.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.341+0000 7f6e709a1640 10 monclient: get_auth_request con 0x7f6e5006da90 auth_method 0 2024-03-29T10:44:49.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.341+0000 7f6e61ffb640 10 client.4661.objecter ms_handle_connect 0x7f6e5006da90 2024-03-29T10:44:49.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.348+0000 7f6e61ffb640 1 -- 172.21.15.84:0/1046148165 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6e6c0cb060 con 0x7f6e64006d80 2024-03-29T10:44:49.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.349+0000 7f6e61ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:49.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.349+0000 7f6e61ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:49.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.664+0000 7f6e721a4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:49.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.664+0000 7f6e721a4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:49.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.664+0000 7f6e721a4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:44:49.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.664+0000 7f6e721a4640 1 -- 172.21.15.84:0/1046148165 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e4c005330 con 0x7f6e64006d80 2024-03-29T10:44:49.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.665+0000 7f6e61ffb640 1 -- 172.21.15.84:0/1046148165 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 1208446409) 0x7f6e6c0fed70 con 0x7f6e64006d80 2024-03-29T10:44:49.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.665+0000 7f6e61ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:49.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.665+0000 7f6e61ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:49.649 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:44:49.649 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":10,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:34:53.505568+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:34:53.505568+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:49.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.669+0000 7f6e721a4640 10 librados: watch_flush enter 2024-03-29T10:44:49.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.669+0000 7f6e721a4640 10 librados: watch_flush exit 2024-03-29T10:44:49.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.669+0000 7f6e721a4640 20 client.4661.objecter shutdown clearing up homeless session... 2024-03-29T10:44:49.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.669+0000 7f6e721a4640 10 client.4661.objecter successfully canceled tick 2024-03-29T10:44:49.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.669+0000 7f6e721a4640 1 -- 172.21.15.84:0/1046148165 >> v1:172.21.15.86:6816/3337225337 conn(0x7f6e5006da90 legacy=0x7f6e5006ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:49.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.669+0000 7f6e721a4640 10 monclient: shutdown 2024-03-29T10:44:49.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.669+0000 7f6e721a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:49.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.669+0000 7f6e721a4640 1 -- 172.21.15.84:0/1046148165 >> v1:172.21.15.84:6790/0 conn(0x7f6e64006d80 legacy=0x7f6e6402cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:49.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.669+0000 7f6e721a4640 1 -- 172.21.15.84:0/1046148165 shutdown_connections 2024-03-29T10:44:49.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.669+0000 7f6e721a4640 1 -- 172.21.15.84:0/1046148165 >> 172.21.15.84:0/1046148165 conn(0x7f6e64005c00 msgr2=0x7f6e640c7fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:49.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.670+0000 7f6e721a4640 1 -- 172.21.15.84:0/1046148165 shutdown_connections 2024-03-29T10:44:49.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.670+0000 7f6e721a4640 1 -- 172.21.15.84:0/1046148165 wait complete. 2024-03-29T10:44:49.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:49.670+0000 7f6e721a4640 1 librados: shutdown 2024-03-29T10:44:49.668 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:44:56.475 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:44:56.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.646+0000 7efc4790e640 10 monclient: get_monmap_and_config 2024-03-29T10:44:56.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.646+0000 7efc4790e640 10 monclient: build_initial_monmap 2024-03-29T10:44:56.629 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.646+0000 7efc4790e640 10 monclient: monmap: 2024-03-29T10:44:56.629 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:44:56.629 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:56.629 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:44:56.647327+0000 2024-03-29T10:44:56.629 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:44:56.647327+0000 2024-03-29T10:44:56.629 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:56.629 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:56.630 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:44:56.630 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:44:56.630 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:44:56.630 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:56.632 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.650+0000 7efc4790e640 1 Processor -- start 2024-03-29T10:44:56.637 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.650+0000 7efc4790e640 1 -- start start 2024-03-29T10:44:56.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.650+0000 7efc4790e640 10 monclient: init 2024-03-29T10:44:56.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.651+0000 7efc4790e640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:56.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.651+0000 7efc4790e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:44:56.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.651+0000 7efc4790e640 10 monclient(hunting): picked mon.noname-c con 0x7efc401422b0 addr v1:172.21.15.86:6789/0 2024-03-29T10:44:56.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.651+0000 7efc4790e640 10 monclient(hunting): picked mon.noname-a con 0x7efc40142690 addr v1:172.21.15.84:6789/0 2024-03-29T10:44:56.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.651+0000 7efc4790e640 10 monclient(hunting): picked mon.noname-b con 0x7efc400b17d0 addr v1:172.21.15.84:6790/0 2024-03-29T10:44:56.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.651+0000 7efc4790e640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc40142c80 con 0x7efc40142690 2024-03-29T10:44:56.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.651+0000 7efc4790e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc400aae40 con 0x7efc401422b0 2024-03-29T10:44:56.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.651+0000 7efc4790e640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc4013fbd0 con 0x7efc400b17d0 2024-03-29T10:44:56.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.651+0000 7efc4790e640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:56.639 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.651+0000 7efc4790e640 10 monclient(hunting): authenticate will time out at 1398.914666s 2024-03-29T10:44:56.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.654+0000 7efc44ea9640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7efc40142690 0x7efc400b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:56887/0 (socket says 172.21.15.84:56887) 2024-03-29T10:44:56.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.654+0000 7efc44ea9640 1 -- 172.21.15.84:0/2422873500 learned_addr learned my addr 172.21.15.84:0/2422873500 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:44:56.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc3ffff640 1 -- 172.21.15.84:0/2422873500 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660800809 0 0) 0x7efc40142c80 con 0x7efc40142690 2024-03-29T10:44:56.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc3ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:44:56.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:56.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:56.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc3ffff640 10 monclient(hunting): my global_id is 4566 2024-03-29T10:44:56.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc3ffff640 1 -- 172.21.15.84:0/2422873500 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc28003520 con 0x7efc40142690 2024-03-29T10:44:56.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc3ffff640 1 -- 172.21.15.84:0/2422873500 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3387772650 0 0) 0x7efc28003520 con 0x7efc40142690 2024-03-29T10:44:56.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc3ffff640 1 -- 172.21.15.84:0/2422873500 >> v1:172.21.15.84:6790/0 conn(0x7efc400b17d0 legacy=0x7efc400b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:56.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc3ffff640 1 -- 172.21.15.84:0/2422873500 >> v1:172.21.15.86:6789/0 conn(0x7efc401422b0 legacy=0x7efc4013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:44:56.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc3ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:44:56.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc3ffff640 1 monclient: found mon.noname-a 2024-03-29T10:44:56.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:56.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc3ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T10:44:56.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc3ffff640 1 -- 172.21.15.84:0/2422873500 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc40151940 con 0x7efc40142690 2024-03-29T10:44:56.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc3ffff640 10 monclient: _finish_auth 0 2024-03-29T10:44:56.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc3ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:44:56.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:56.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc3ffff640 1 -- 172.21.15.84:0/2422873500 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7efc34001690 con 0x7efc40142690 2024-03-29T10:44:56.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:56.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc3ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:56.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc3ffff640 10 monclient: dump: 2024-03-29T10:44:56.641 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:56.641 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:56.641 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:56.641 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:56.641 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:56.642 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:56.642 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:56.642 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:56.642 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:56.642 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:56.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc4790e640 5 monclient: authenticate success, global_id 4566 2024-03-29T10:44:56.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc4790e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:44:56.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc3ffff640 1 -- 172.21.15.84:0/2422873500 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc340017f0 con 0x7efc40142690 2024-03-29T10:44:56.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc3ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:56.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc3ffff640 1 -- 172.21.15.84:0/2422873500 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7efc34001c40 con 0x7efc40142690 2024-03-29T10:44:56.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:56.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc3ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:56.642 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc3ffff640 10 monclient: dump: 2024-03-29T10:44:56.642 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:56.643 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:56.643 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:56.643 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:56.643 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:56.643 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:56.643 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:56.643 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:56.643 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:56.643 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:56.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc4790e640 10 monclient: get_monmap_and_config success 2024-03-29T10:44:56.643 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.655+0000 7efc4790e640 4 set_mon_vals no callback set 2024-03-29T10:44:56.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.656+0000 7efc4790e640 10 monclient: shutdown 2024-03-29T10:44:56.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.656+0000 7efc4790e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:56.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.656+0000 7efc4790e640 1 -- 172.21.15.84:0/2422873500 >> v1:172.21.15.84:6789/0 conn(0x7efc40142690 legacy=0x7efc400b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:56.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.658+0000 7efc4790e640 1 -- 172.21.15.84:0/2422873500 shutdown_connections 2024-03-29T10:44:56.645 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.658+0000 7efc4790e640 1 -- 172.21.15.84:0/2422873500 >> 172.21.15.84:0/2422873500 conn(0x7efc40066a00 msgr2=0x7efc40066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:56.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.658+0000 7efc4790e640 1 -- 172.21.15.84:0/2422873500 shutdown_connections 2024-03-29T10:44:56.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.659+0000 7efc4790e640 1 -- 172.21.15.84:0/2422873500 wait complete. 2024-03-29T10:44:56.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.660+0000 7efc4790e640 10 monclient: build_initial_monmap 2024-03-29T10:44:56.646 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.660+0000 7efc4790e640 10 monclient: monmap: 2024-03-29T10:44:56.646 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:44:56.646 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:44:56.646 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:44:56.646 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:44:56.646 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:44:56.646 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:56.646 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:44:56.647 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:44:56.647 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:44:56.647 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:56.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.661+0000 7efc4790e640 1 librados: starting msgr at 2024-03-29T10:44:56.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.662+0000 7efc4790e640 1 librados: starting objecter 2024-03-29T10:44:56.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.662+0000 7efc4790e640 1 Processor -- start 2024-03-29T10:44:56.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.662+0000 7efc4790e640 1 -- start start 2024-03-29T10:44:56.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.662+0000 7efc4790e640 1 librados: setting wanted keys 2024-03-29T10:44:56.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.663+0000 7efc4790e640 1 librados: calling monclient init 2024-03-29T10:44:56.647 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.663+0000 7efc4790e640 10 monclient: init 2024-03-29T10:44:56.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.664+0000 7efc4790e640 10 monclient: _reopen_session rank -1 2024-03-29T10:44:56.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.664+0000 7efc4790e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:44:56.648 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.664+0000 7efc4790e640 10 monclient(hunting): picked mon.noname-c con 0x7efc400b17d0 addr v1:172.21.15.84:6790/0 2024-03-29T10:44:56.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.666+0000 7efc44ea9640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7efc401422b0 0x7efc400b6e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:57133/0 (socket says 172.21.15.84:57133) 2024-03-29T10:44:56.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.667+0000 7efc44ea9640 1 -- 172.21.15.84:0/2349555845 learned_addr learned my addr 172.21.15.84:0/2349555845 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:44:56.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.667+0000 7efc4790e640 10 monclient(hunting): picked mon.noname-a con 0x7efc401422b0 addr v1:172.21.15.84:6789/0 2024-03-29T10:44:56.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.667+0000 7efc3dffb640 10 client.?.objecter ms_handle_connect 0x7efc401422b0 2024-03-29T10:44:56.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.667+0000 7efc3dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:56.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.667+0000 7efc3dffb640 10 client.?.objecter ms_handle_connect 0x7efc400b17d0 2024-03-29T10:44:56.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.667+0000 7efc3dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:44:56.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.667+0000 7efc4790e640 10 monclient(hunting): picked mon.noname-b con 0x7efc40142690 addr v1:172.21.15.86:6789/0 2024-03-29T10:44:56.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.667+0000 7efc4790e640 1 -- 172.21.15.84:0/2349555845 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc4014e410 con 0x7efc401422b0 2024-03-29T10:44:56.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.667+0000 7efc4790e640 1 -- 172.21.15.84:0/2349555845 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc4006f0d0 con 0x7efc40142690 2024-03-29T10:44:56.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.667+0000 7efc4790e640 1 -- 172.21.15.84:0/2349555845 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc4006f3b0 con 0x7efc400b17d0 2024-03-29T10:44:56.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.667+0000 7efc4790e640 10 monclient(hunting): _renew_subs 2024-03-29T10:44:56.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.667+0000 7efc4790e640 10 monclient(hunting): authenticate will time out at 1398.930857s 2024-03-29T10:44:56.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.667+0000 7efc3dffb640 1 -- 172.21.15.84:0/2349555845 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040042759 0 0) 0x7efc4014e410 con 0x7efc401422b0 2024-03-29T10:44:56.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.667+0000 7efc3dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:44:56.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.667+0000 7efc3dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:56.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.667+0000 7efc3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:56.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.667+0000 7efc3dffb640 10 monclient(hunting): my global_id is 4569 2024-03-29T10:44:56.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.667+0000 7efc3dffb640 1 -- 172.21.15.84:0/2349555845 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc2c004eb0 con 0x7efc401422b0 2024-03-29T10:44:56.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.669+0000 7efc3dffb640 1 -- 172.21.15.84:0/2349555845 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058635393 0 0) 0x7efc4006f3b0 con 0x7efc400b17d0 2024-03-29T10:44:56.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.669+0000 7efc3dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:44:56.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.669+0000 7efc3dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:44:56.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.669+0000 7efc3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:44:56.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.669+0000 7efc3dffb640 10 monclient(hunting): my global_id is 4667 2024-03-29T10:44:56.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.669+0000 7efc3dffb640 1 -- 172.21.15.84:0/2349555845 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc4014e410 con 0x7efc400b17d0 2024-03-29T10:44:56.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.669+0000 7efc3dffb640 1 -- 172.21.15.84:0/2349555845 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 266058113 0 0) 0x7efc2c004eb0 con 0x7efc401422b0 2024-03-29T10:44:56.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.669+0000 7efc3dffb640 1 -- 172.21.15.84:0/2349555845 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc4006f3b0 con 0x7efc401422b0 2024-03-29T10:44:56.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.669+0000 7efc3dffb640 1 -- 172.21.15.84:0/2349555845 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7efc34001b80 con 0x7efc401422b0 2024-03-29T10:44:56.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.669+0000 7efc3dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:56.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.669+0000 7efc3dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:44:56.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.669+0000 7efc3dffb640 10 monclient(hunting): dump: 2024-03-29T10:44:56.656 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:56.656 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:56.656 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:56.656 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:56.656 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:56.656 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:56.656 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:56.656 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:56.656 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:56.656 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:56.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.669+0000 7efc3dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:44:56.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.670+0000 7efc3dffb640 1 -- 172.21.15.84:0/2349555845 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2172014217 0 0) 0x7efc4006f3b0 con 0x7efc401422b0 2024-03-29T10:44:56.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.670+0000 7efc3dffb640 1 -- 172.21.15.84:0/2349555845 >> v1:172.21.15.84:6790/0 conn(0x7efc400b17d0 legacy=0x7efc4014f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:56.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.670+0000 7efc3dffb640 1 -- 172.21.15.84:0/2349555845 >> v1:172.21.15.86:6789/0 conn(0x7efc40142690 legacy=0x7efc401ddc40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:44:56.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.670+0000 7efc3dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:44:56.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.670+0000 7efc3dffb640 1 monclient: found mon.a 2024-03-29T10:44:56.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.670+0000 7efc3dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:44:56.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.670+0000 7efc3dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:44:56.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.670+0000 7efc3dffb640 1 -- 172.21.15.84:0/2349555845 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc4006f690 con 0x7efc401422b0 2024-03-29T10:44:56.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.670+0000 7efc3dffb640 10 monclient: _finish_auth 0 2024-03-29T10:44:56.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.670+0000 7efc3dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:44:56.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.670+0000 7efc3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:44:56.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.670+0000 7efc4790e640 5 monclient: authenticate success, global_id 4569 2024-03-29T10:44:56.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.670+0000 7efc4790e640 10 monclient: _renew_subs 2024-03-29T10:44:56.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.670+0000 7efc4790e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:44:56.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.670+0000 7efc4790e640 1 -- 172.21.15.84:0/2349555845 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc4006e420 con 0x7efc401422b0 2024-03-29T10:44:56.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.670+0000 7efc4790e640 10 client.4569.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:44:56.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.670+0000 7efc4790e640 10 monclient: _renew_subs 2024-03-29T10:44:56.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.670+0000 7efc4790e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:44:56.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.670+0000 7efc4790e640 1 -- 172.21.15.84:0/2349555845 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc4006e050 con 0x7efc401422b0 2024-03-29T10:44:56.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.670+0000 7efc4790e640 1 librados: init done 2024-03-29T10:44:56.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.671+0000 7efc3dffb640 1 -- 172.21.15.84:0/2349555845 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc34004a90 con 0x7efc401422b0 2024-03-29T10:44:56.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.671+0000 7efc3dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:44:56.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.671+0000 7efc3dffb640 1 -- 172.21.15.84:0/2349555845 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7efc34001270 con 0x7efc401422b0 2024-03-29T10:44:56.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.671+0000 7efc3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:44:56.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.671+0000 7efc3dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:44:56.658 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.671+0000 7efc3dffb640 10 monclient: dump: 2024-03-29T10:44:56.658 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:44:56.658 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:44:56.658 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:56.658 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:44:56.658 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:44:56.659 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:44:56.659 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:44:56.659 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:44:56.659 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:44:56.659 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:44:56.659 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.671+0000 7efc3f7fe640 4 set_mon_vals no callback set 2024-03-29T10:44:56.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.673+0000 7efc3dffb640 1 -- 172.21.15.84:0/2349555845 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7efc3401ac10 con 0x7efc401422b0 2024-03-29T10:44:56.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.674+0000 7efc3dffb640 1 -- 172.21.15.84:0/2349555845 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1523682299 0 0) 0x7efc34084b50 con 0x7efc401422b0 2024-03-29T10:44:56.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.674+0000 7efc3dffb640 10 client.4569.objecter ms_dispatch 0x7efc40000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:44:56.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.674+0000 7efc3dffb640 3 client.4569.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:44:56.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.674+0000 7efc3dffb640 3 client.4569.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:44:56.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.674+0000 7efc3dffb640 20 client.4569.objecter dump_active .. 0 homeless 2024-03-29T10:44:56.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.674+0000 7efc456aa640 10 monclient: get_auth_request con 0x7efc2c06dcd0 auth_method 0 2024-03-29T10:44:56.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.674+0000 7efc3dffb640 10 client.4569.objecter ms_handle_connect 0x7efc2c06dcd0 2024-03-29T10:44:56.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.675+0000 7efc4790e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:56.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.675+0000 7efc4790e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:56.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.675+0000 7efc4790e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:44:56.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.675+0000 7efc4790e640 1 -- 172.21.15.84:0/2349555845 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc20005080 con 0x7efc401422b0 2024-03-29T10:44:56.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.683+0000 7efc3dffb640 1 -- 172.21.15.84:0/2349555845 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efc34050430 con 0x7efc401422b0 2024-03-29T10:44:56.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.683+0000 7efc3dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:44:56.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:56.683+0000 7efc3dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:44:56.987 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:57.004+0000 7efc4790e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:56.987 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:57.004+0000 7efc4790e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:56.987 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:57.004+0000 7efc4790e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:44:56.987 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:57.004+0000 7efc4790e640 1 -- 172.21.15.84:0/2349555845 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc20005330 con 0x7efc401422b0 2024-03-29T10:44:56.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:57.005+0000 7efc3dffb640 1 -- 172.21.15.84:0/2349555845 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 879675878) 0x7efc34052360 con 0x7efc401422b0 2024-03-29T10:44:56.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:57.005+0000 7efc3dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:44:56.988 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:57.005+0000 7efc3dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:44:56.988 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:44:56.988 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":11,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:44:56.255219+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:56.255219+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:44:56.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:57.010+0000 7efc4790e640 10 librados: watch_flush enter 2024-03-29T10:44:56.993 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:57.010+0000 7efc4790e640 10 librados: watch_flush exit 2024-03-29T10:44:56.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:57.010+0000 7efc4790e640 20 client.4569.objecter shutdown clearing up homeless session... 2024-03-29T10:44:56.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:57.010+0000 7efc4790e640 10 client.4569.objecter successfully canceled tick 2024-03-29T10:44:56.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:57.010+0000 7efc4790e640 1 -- 172.21.15.84:0/2349555845 >> v1:172.21.15.86:6816/3337225337 conn(0x7efc2c06dcd0 legacy=0x7efc2c070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:56.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:57.010+0000 7efc4790e640 10 monclient: shutdown 2024-03-29T10:44:56.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:57.010+0000 7efc4790e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:44:56.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:57.010+0000 7efc4790e640 1 -- 172.21.15.84:0/2349555845 >> v1:172.21.15.84:6789/0 conn(0x7efc401422b0 legacy=0x7efc400b6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:44:56.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:57.010+0000 7efc4790e640 1 -- 172.21.15.84:0/2349555845 shutdown_connections 2024-03-29T10:44:56.994 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:57.010+0000 7efc4790e640 1 -- 172.21.15.84:0/2349555845 >> 172.21.15.84:0/2349555845 conn(0x7efc40066a00 msgr2=0x7efc400b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:44:56.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:57.012+0000 7efc4790e640 1 -- 172.21.15.84:0/2349555845 shutdown_connections 2024-03-29T10:44:56.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:57.012+0000 7efc4790e640 1 -- 172.21.15.84:0/2349555845 wait complete. 2024-03-29T10:44:56.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:44:57.012+0000 7efc4790e640 1 librados: shutdown 2024-03-29T10:44:57.011 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:03.818 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:03.966 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.983+0000 7f11e10fb640 10 monclient: get_monmap_and_config 2024-03-29T10:45:03.966 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.983+0000 7f11e10fb640 10 monclient: build_initial_monmap 2024-03-29T10:45:03.966 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.983+0000 7f11e10fb640 10 monclient: monmap: 2024-03-29T10:45:03.966 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:45:03.967 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:03.967 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:45:03.984453+0000 2024-03-29T10:45:03.967 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:45:03.984453+0000 2024-03-29T10:45:03.967 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:03.967 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:03.967 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:45:03.967 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:45:03.967 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:45:03.967 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:03.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.985+0000 7f11e10fb640 1 Processor -- start 2024-03-29T10:45:03.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.986+0000 7f11e10fb640 1 -- start start 2024-03-29T10:45:03.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.986+0000 7f11e10fb640 10 monclient: init 2024-03-29T10:45:03.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.986+0000 7f11e10fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:03.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.986+0000 7f11e10fb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:45:03.969 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.986+0000 7f11e10fb640 10 monclient(hunting): picked mon.noname-a con 0x7f11d4006890 addr v1:172.21.15.84:6789/0 2024-03-29T10:45:03.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.987+0000 7f11e10fb640 10 monclient(hunting): picked mon.noname-c con 0x7f11d4006c70 addr v1:172.21.15.86:6789/0 2024-03-29T10:45:03.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.987+0000 7f11e10fb640 10 monclient(hunting): picked mon.noname-b con 0x7f11d402b4c0 addr v1:172.21.15.84:6790/0 2024-03-29T10:45:03.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.987+0000 7f11e10fb640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d402c830 con 0x7f11d4006890 2024-03-29T10:45:03.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.987+0000 7f11e10fb640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d402afa0 con 0x7f11d4006c70 2024-03-29T10:45:03.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.987+0000 7f11e10fb640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d402aba0 con 0x7f11d402b4c0 2024-03-29T10:45:03.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.987+0000 7f11e10fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:03.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.987+0000 7f11e10fb640 10 monclient(hunting): authenticate will time out at 1406.250275s 2024-03-29T10:45:03.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.987+0000 7f11e08fa640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f11d402b4c0 0x7f11d40cf0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:55136/0 (socket says 172.21.15.84:55136) 2024-03-29T10:45:03.970 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.987+0000 7f11e08fa640 1 -- 172.21.15.84:0/222915535 learned_addr learned my addr 172.21.15.84:0/222915535 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:45:03.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.990+0000 7f11daffd640 1 -- 172.21.15.84:0/222915535 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3010573422 0 0) 0x7f11d402aba0 con 0x7f11d402b4c0 2024-03-29T10:45:03.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.990+0000 7f11daffd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:03.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.990+0000 7f11daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:03.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.990+0000 7f11daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:03.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.990+0000 7f11daffd640 10 monclient(hunting): my global_id is 4670 2024-03-29T10:45:03.973 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.990+0000 7f11daffd640 1 -- 172.21.15.84:0/222915535 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11cc003520 con 0x7f11d402b4c0 2024-03-29T10:45:03.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.990+0000 7f11daffd640 1 -- 172.21.15.84:0/222915535 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3654687885 0 0) 0x7f11d402c830 con 0x7f11d4006890 2024-03-29T10:45:03.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11daffd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:03.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:03.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:03.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11daffd640 10 monclient(hunting): my global_id is 4572 2024-03-29T10:45:03.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11daffd640 1 -- 172.21.15.84:0/222915535 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11d402aba0 con 0x7f11d4006890 2024-03-29T10:45:03.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11daffd640 1 -- 172.21.15.84:0/222915535 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3130866376 0 0) 0x7f11cc003520 con 0x7f11d402b4c0 2024-03-29T10:45:03.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11daffd640 1 -- 172.21.15.84:0/222915535 >> v1:172.21.15.86:6789/0 conn(0x7f11d4006c70 legacy=0x7f11d40c6d40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:45:03.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11daffd640 1 -- 172.21.15.84:0/222915535 >> v1:172.21.15.84:6789/0 conn(0x7f11d4006890 legacy=0x7f11d40c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:03.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11daffd640 10 monclient: _finish_hunting 0 2024-03-29T10:45:03.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11daffd640 1 monclient: found mon.noname-b 2024-03-29T10:45:03.974 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:03.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11daffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:45:03.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11daffd640 1 -- 172.21.15.84:0/222915535 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d402ad30 con 0x7f11d402b4c0 2024-03-29T10:45:03.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11daffd640 10 monclient: _finish_auth 0 2024-03-29T10:45:03.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11daffd640 10 monclient: _check_auth_tickets 2024-03-29T10:45:03.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:03.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11daffd640 1 -- 172.21.15.84:0/222915535 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f11c8001610 con 0x7f11d402b4c0 2024-03-29T10:45:03.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:03.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11daffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:03.975 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11daffd640 10 monclient: dump: 2024-03-29T10:45:03.975 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:03.975 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:03.975 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:03.975 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:03.975 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:03.975 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:03.976 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:03.976 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:03.976 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:03.976 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:03.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11e10fb640 5 monclient: authenticate success, global_id 4670 2024-03-29T10:45:03.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11e10fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:03.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11daffd640 1 -- 172.21.15.84:0/222915535 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11c8001770 con 0x7f11d402b4c0 2024-03-29T10:45:03.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:03.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11daffd640 1 -- 172.21.15.84:0/222915535 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f11c8001b80 con 0x7f11d402b4c0 2024-03-29T10:45:03.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11e10fb640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:03.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11e10fb640 4 set_mon_vals no callback set 2024-03-29T10:45:03.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:03.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:03.976 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.991+0000 7f11daffd640 10 monclient: dump: 2024-03-29T10:45:03.977 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:03.977 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:03.977 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:03.977 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:03.977 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:03.977 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:03.977 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:03.977 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:03.977 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:03.977 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:03.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.992+0000 7f11e10fb640 10 monclient: shutdown 2024-03-29T10:45:03.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.992+0000 7f11e10fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:03.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.992+0000 7f11e10fb640 1 -- 172.21.15.84:0/222915535 >> v1:172.21.15.84:6790/0 conn(0x7f11d402b4c0 legacy=0x7f11d40cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:03.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.992+0000 7f11e10fb640 1 -- 172.21.15.84:0/222915535 shutdown_connections 2024-03-29T10:45:03.978 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.992+0000 7f11e10fb640 1 -- 172.21.15.84:0/222915535 >> 172.21.15.84:0/222915535 conn(0x7f11d40058f0 msgr2=0x7f11d4005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:03.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.993+0000 7f11e10fb640 1 -- 172.21.15.84:0/222915535 shutdown_connections 2024-03-29T10:45:03.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.994+0000 7f11e10fb640 1 -- 172.21.15.84:0/222915535 wait complete. 2024-03-29T10:45:03.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.995+0000 7f11e10fb640 10 monclient: build_initial_monmap 2024-03-29T10:45:03.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.995+0000 7f11e10fb640 10 monclient: monmap: 2024-03-29T10:45:03.979 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:45:03.979 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:03.979 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:45:03.979 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:45:03.979 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:03.979 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:03.979 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:45:03.979 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:45:03.979 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:45:03.979 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:03.979 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.995+0000 7f11e10fb640 1 librados: starting msgr at 2024-03-29T10:45:03.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.996+0000 7f11e10fb640 1 librados: starting objecter 2024-03-29T10:45:03.980 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.996+0000 7f11e10fb640 1 Processor -- start 2024-03-29T10:45:03.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.998+0000 7f11e10fb640 1 -- start start 2024-03-29T10:45:03.981 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.999+0000 7f11e10fb640 1 librados: setting wanted keys 2024-03-29T10:45:03.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.999+0000 7f11e10fb640 1 librados: calling monclient init 2024-03-29T10:45:03.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:03.999+0000 7f11e10fb640 10 monclient: init 2024-03-29T10:45:03.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.000+0000 7f11e10fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:03.982 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.000+0000 7f11e10fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:45:03.983 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.000+0000 7f11e10fb640 10 monclient(hunting): picked mon.noname-b con 0x7f11d4006890 addr v1:172.21.15.86:6789/0 2024-03-29T10:45:03.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.000+0000 7f11e10fb640 10 monclient(hunting): picked mon.noname-a con 0x7f11d402b4c0 addr v1:172.21.15.84:6789/0 2024-03-29T10:45:03.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.000+0000 7f11e10fb640 10 monclient(hunting): picked mon.noname-c con 0x7f11d40c15c0 addr v1:172.21.15.84:6790/0 2024-03-29T10:45:03.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.000+0000 7f11e08fa640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f11d40c15c0 0x7f11d40bc150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:58898/0 (socket says 172.21.15.84:58898) 2024-03-29T10:45:03.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.000+0000 7f11e08fa640 1 -- 172.21.15.84:0/241208099 learned_addr learned my addr 172.21.15.84:0/241208099 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:45:03.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.000+0000 7f11e10fb640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d40bc860 con 0x7f11d402b4c0 2024-03-29T10:45:03.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.000+0000 7f11e10fb640 1 -- 172.21.15.84:0/241208099 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d402de00 con 0x7f11d4006890 2024-03-29T10:45:03.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.000+0000 7f11e10fb640 1 -- 172.21.15.84:0/241208099 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d4158830 con 0x7f11d40c15c0 2024-03-29T10:45:03.984 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.000+0000 7f11e10fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:03.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.000+0000 7f11e10fb640 10 monclient(hunting): authenticate will time out at 1406.263758s 2024-03-29T10:45:03.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.001+0000 7f11d8ff9640 10 client.?.objecter ms_handle_connect 0x7f11d40c15c0 2024-03-29T10:45:03.985 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.001+0000 7f11d8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:03.987 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.003+0000 7f11d8ff9640 1 -- 172.21.15.84:0/241208099 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212609707 0 0) 0x7f11d4158830 con 0x7f11d40c15c0 2024-03-29T10:45:03.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.003+0000 7f11d8ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:45:03.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.003+0000 7f11d8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:03.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.003+0000 7f11d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:03.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.003+0000 7f11d8ff9640 10 monclient(hunting): my global_id is 4673 2024-03-29T10:45:03.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.003+0000 7f11d8ff9640 1 -- 172.21.15.84:0/241208099 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11c0004eb0 con 0x7f11d40c15c0 2024-03-29T10:45:03.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.004+0000 7f11d8ff9640 1 -- 172.21.15.84:0/241208099 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3479261683 0 0) 0x7f11c0004eb0 con 0x7f11d40c15c0 2024-03-29T10:45:03.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.004+0000 7f11d8ff9640 1 -- 172.21.15.84:0/241208099 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11d4158830 con 0x7f11d40c15c0 2024-03-29T10:45:03.989 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.004+0000 7f11d8ff9640 1 -- 172.21.15.84:0/241208099 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f11c8001a80 con 0x7f11d40c15c0 2024-03-29T10:45:03.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.004+0000 7f11d8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:03.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.004+0000 7f11d8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:03.990 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.004+0000 7f11d8ff9640 10 monclient(hunting): dump: 2024-03-29T10:45:03.990 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:03.990 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:03.990 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:03.990 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:03.990 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:03.990 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:03.990 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:03.990 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:03.990 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:03.990 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:03.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.004+0000 7f11d8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:03.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.008+0000 7f11d8ff9640 10 client.?.objecter ms_handle_connect 0x7f11d402b4c0 2024-03-29T10:45:03.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.008+0000 7f11d8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:03.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.008+0000 7f11d8ff9640 1 -- 172.21.15.84:0/241208099 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3775760598 0 0) 0x7f11d4158830 con 0x7f11d40c15c0 2024-03-29T10:45:03.991 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.008+0000 7f11d8ff9640 1 -- 172.21.15.84:0/241208099 >> v1:172.21.15.86:6789/0 conn(0x7f11d4006890 legacy=0x7f11d4158d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:45:03.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.010+0000 7f11d8ff9640 1 -- 172.21.15.84:0/241208099 >> v1:172.21.15.84:6789/0 conn(0x7f11d402b4c0 legacy=0x7f11d40bba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:03.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.010+0000 7f11d8ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:45:03.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.010+0000 7f11d8ff9640 1 monclient: found mon.c 2024-03-29T10:45:03.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.010+0000 7f11d8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:03.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.010+0000 7f11d8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:45:03.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.010+0000 7f11d8ff9640 1 -- 172.21.15.84:0/241208099 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d40c3850 con 0x7f11d40c15c0 2024-03-29T10:45:03.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.010+0000 7f11d8ff9640 10 monclient: _finish_auth 0 2024-03-29T10:45:03.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.010+0000 7f11d8ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:45:03.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.010+0000 7f11d8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:03.995 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.010+0000 7f11e10fb640 5 monclient: authenticate success, global_id 4673 2024-03-29T10:45:03.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.010+0000 7f11e10fb640 10 monclient: _renew_subs 2024-03-29T10:45:03.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.010+0000 7f11e10fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:45:03.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.010+0000 7f11e10fb640 1 -- 172.21.15.84:0/241208099 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d4159460 con 0x7f11d40c15c0 2024-03-29T10:45:03.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.010+0000 7f11e10fb640 10 client.4673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:03.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.010+0000 7f11e10fb640 10 monclient: _renew_subs 2024-03-29T10:45:03.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.010+0000 7f11e10fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:45:03.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.010+0000 7f11e10fb640 1 -- 172.21.15.84:0/241208099 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d40c4510 con 0x7f11d40c15c0 2024-03-29T10:45:03.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.010+0000 7f11e10fb640 1 librados: init done 2024-03-29T10:45:03.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.010+0000 7f11d8ff9640 1 -- 172.21.15.84:0/241208099 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11c8004a90 con 0x7f11d40c15c0 2024-03-29T10:45:03.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.010+0000 7f11d8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:03.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.011+0000 7f11d8ff9640 1 -- 172.21.15.84:0/241208099 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f11c8004ee0 con 0x7f11d40c15c0 2024-03-29T10:45:03.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.011+0000 7f11d8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:03.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.011+0000 7f11d8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:03.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.011+0000 7f11da7fc640 4 set_mon_vals no callback set 2024-03-29T10:45:03.996 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.012+0000 7f11e10fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:03.997 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.012+0000 7f11d8ff9640 10 monclient: dump: 2024-03-29T10:45:03.997 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:03.997 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:03.997 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:03.997 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:03.997 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:03.997 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:03.997 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:03.997 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:03.997 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:03.997 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:03.997 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.012+0000 7f11e10fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:03.997 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.012+0000 7f11e10fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:45:03.997 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.012+0000 7f11e10fb640 1 -- 172.21.15.84:0/241208099 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11bc005080 con 0x7f11d40c15c0 2024-03-29T10:45:03.998 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.013+0000 7f11d8ff9640 1 -- 172.21.15.84:0/241208099 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f11c8005040 con 0x7f11d40c15c0 2024-03-29T10:45:03.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.015+0000 7f11dbfff640 10 monclient: get_auth_request con 0x7f11c006dae0 auth_method 0 2024-03-29T10:45:03.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.015+0000 7f11d8ff9640 1 -- 172.21.15.84:0/241208099 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1523682299 0 0) 0x7f11c8084c10 con 0x7f11d40c15c0 2024-03-29T10:45:03.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.015+0000 7f11d8ff9640 10 client.4673.objecter ms_dispatch 0x7f11dc000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:45:03.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.015+0000 7f11d8ff9640 3 client.4673.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:45:03.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.015+0000 7f11d8ff9640 3 client.4673.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:45:03.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.015+0000 7f11d8ff9640 20 client.4673.objecter dump_active .. 0 homeless 2024-03-29T10:45:03.999 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.015+0000 7f11d8ff9640 10 client.4673.objecter ms_handle_connect 0x7f11c006dae0 2024-03-29T10:45:04.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.022+0000 7f11d8ff9640 1 -- 172.21.15.84:0/241208099 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f11c80504f0 con 0x7f11d40c15c0 2024-03-29T10:45:04.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.022+0000 7f11d8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:04.005 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.022+0000 7f11d8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:04.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.345+0000 7f11e10fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:04.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.345+0000 7f11e10fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:04.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.345+0000 7f11e10fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:45:04.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.345+0000 7f11e10fb640 1 -- 172.21.15.84:0/241208099 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11bc005330 con 0x7f11d40c15c0 2024-03-29T10:45:04.331 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.348+0000 7f11d8ff9640 1 -- 172.21.15.84:0/241208099 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 879675878) 0x7f11c8052420 con 0x7f11d40c15c0 2024-03-29T10:45:04.331 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.348+0000 7f11d8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:04.331 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.348+0000 7f11d8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:04.331 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:45:04.331 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":11,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:44:56.255219+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:56.255219+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:04.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.352+0000 7f11e10fb640 10 librados: watch_flush enter 2024-03-29T10:45:04.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.352+0000 7f11e10fb640 10 librados: watch_flush exit 2024-03-29T10:45:04.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.352+0000 7f11e10fb640 20 client.4673.objecter shutdown clearing up homeless session... 2024-03-29T10:45:04.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.352+0000 7f11e10fb640 10 client.4673.objecter successfully canceled tick 2024-03-29T10:45:04.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.352+0000 7f11e10fb640 1 -- 172.21.15.84:0/241208099 >> v1:172.21.15.86:6816/3337225337 conn(0x7f11c006dae0 legacy=0x7f11c006ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:04.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.352+0000 7f11e10fb640 10 monclient: shutdown 2024-03-29T10:45:04.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.352+0000 7f11e10fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:04.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.352+0000 7f11e10fb640 1 -- 172.21.15.84:0/241208099 >> v1:172.21.15.84:6790/0 conn(0x7f11d40c15c0 legacy=0x7f11d40bc150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:04.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.352+0000 7f11e10fb640 1 -- 172.21.15.84:0/241208099 shutdown_connections 2024-03-29T10:45:04.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.352+0000 7f11e10fb640 1 -- 172.21.15.84:0/241208099 >> 172.21.15.84:0/241208099 conn(0x7f11d40058f0 msgr2=0x7f11d4038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:04.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.352+0000 7f11e10fb640 1 -- 172.21.15.84:0/241208099 shutdown_connections 2024-03-29T10:45:04.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.352+0000 7f11e10fb640 1 -- 172.21.15.84:0/241208099 wait complete. 2024-03-29T10:45:04.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:04.352+0000 7f11e10fb640 1 librados: shutdown 2024-03-29T10:45:04.352 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:11.159 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:11.311 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.327+0000 7f0b2cac8640 10 monclient: get_monmap_and_config 2024-03-29T10:45:11.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.327+0000 7f0b2cac8640 10 monclient: build_initial_monmap 2024-03-29T10:45:11.312 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.327+0000 7f0b2cac8640 10 monclient: monmap: 2024-03-29T10:45:11.312 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:45:11.312 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:11.312 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:45:11.328781+0000 2024-03-29T10:45:11.312 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:45:11.328781+0000 2024-03-29T10:45:11.312 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:11.312 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:11.312 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:45:11.312 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:45:11.312 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:45:11.313 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:11.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.329+0000 7f0b2cac8640 1 Processor -- start 2024-03-29T10:45:11.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.329+0000 7f0b2cac8640 1 -- start start 2024-03-29T10:45:11.313 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.329+0000 7f0b2cac8640 10 monclient: init 2024-03-29T10:45:11.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.331+0000 7f0b2cac8640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:11.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.331+0000 7f0b2cac8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:45:11.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.331+0000 7f0b2cac8640 10 monclient(hunting): picked mon.noname-b con 0x7f0b280b4100 addr v1:172.21.15.84:6790/0 2024-03-29T10:45:11.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.331+0000 7f0b2cac8640 10 monclient(hunting): picked mon.noname-c con 0x7f0b2813ee90 addr v1:172.21.15.86:6789/0 2024-03-29T10:45:11.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.331+0000 7f0b2cac8640 10 monclient(hunting): picked mon.noname-a con 0x7f0b280af900 addr v1:172.21.15.84:6789/0 2024-03-29T10:45:11.318 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.331+0000 7f0b2cac8640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b280afd10 con 0x7f0b280af900 2024-03-29T10:45:11.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.331+0000 7f0b2cac8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b280af6f0 con 0x7f0b2813ee90 2024-03-29T10:45:11.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.331+0000 7f0b2cac8640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b280aaf60 con 0x7f0b280b4100 2024-03-29T10:45:11.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.331+0000 7f0b2cac8640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:11.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.331+0000 7f0b2cac8640 10 monclient(hunting): authenticate will time out at 1413.594962s 2024-03-29T10:45:11.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.335+0000 7f0b25d9b640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f0b2813ee90 0x7f0b280adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:47927/0 (socket says 172.21.15.84:47927) 2024-03-29T10:45:11.319 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.335+0000 7f0b25d9b640 1 -- 172.21.15.84:0/395326532 learned_addr learned my addr 172.21.15.84:0/395326532 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:45:11.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.336+0000 7f0b2559a640 1 -- 172.21.15.84:0/395326532 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3364443247 0 0) 0x7f0b280aaf60 con 0x7f0b280b4100 2024-03-29T10:45:11.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.336+0000 7f0b2559a640 10 monclient(hunting): _negotiate 2024-03-29T10:45:11.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.336+0000 7f0b2559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:11.321 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.336+0000 7f0b2559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:11.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.336+0000 7f0b2559a640 10 monclient(hunting): my global_id is 4676 2024-03-29T10:45:11.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.336+0000 7f0b2559a640 1 -- 172.21.15.84:0/395326532 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b10003520 con 0x7f0b280b4100 2024-03-29T10:45:11.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.336+0000 7f0b2559a640 1 -- 172.21.15.84:0/395326532 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2691121866 0 0) 0x7f0b280af6f0 con 0x7f0b2813ee90 2024-03-29T10:45:11.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.336+0000 7f0b2559a640 10 monclient(hunting): _negotiate 2024-03-29T10:45:11.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.336+0000 7f0b2559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:11.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.336+0000 7f0b2559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:11.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.336+0000 7f0b2559a640 10 monclient(hunting): my global_id is 4522 2024-03-29T10:45:11.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.336+0000 7f0b2559a640 1 -- 172.21.15.84:0/395326532 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b10003840 con 0x7f0b2813ee90 2024-03-29T10:45:11.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.338+0000 7f0b2559a640 1 -- 172.21.15.84:0/395326532 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3108160994 0 0) 0x7f0b10003840 con 0x7f0b2813ee90 2024-03-29T10:45:11.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.338+0000 7f0b2559a640 1 -- 172.21.15.84:0/395326532 >> v1:172.21.15.84:6790/0 conn(0x7f0b280b4100 legacy=0x7f0b280a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:11.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.338+0000 7f0b2559a640 1 -- 172.21.15.84:0/395326532 >> v1:172.21.15.84:6789/0 conn(0x7f0b280af900 legacy=0x7f0b2813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:11.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.338+0000 7f0b2559a640 10 monclient: _finish_hunting 0 2024-03-29T10:45:11.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.338+0000 7f0b2559a640 1 monclient: found mon.noname-c 2024-03-29T10:45:11.322 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.338+0000 7f0b2559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:11.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.338+0000 7f0b2559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.86:6789/0 2024-03-29T10:45:11.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.338+0000 7f0b2559a640 1 -- 172.21.15.84:0/395326532 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2813e970 con 0x7f0b2813ee90 2024-03-29T10:45:11.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.338+0000 7f0b2559a640 10 monclient: _finish_auth 0 2024-03-29T10:45:11.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.338+0000 7f0b2559a640 10 monclient: _check_auth_tickets 2024-03-29T10:45:11.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.338+0000 7f0b2559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:11.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.338+0000 7f0b2559a640 1 -- 172.21.15.84:0/395326532 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0b180015f0 con 0x7f0b2813ee90 2024-03-29T10:45:11.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.338+0000 7f0b2559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:11.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.338+0000 7f0b2559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:11.323 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.338+0000 7f0b2559a640 10 monclient: dump: 2024-03-29T10:45:11.323 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:11.323 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:11.323 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:11.323 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:11.324 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:11.324 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:11.324 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:11.324 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:11.324 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:11.324 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:11.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.339+0000 7f0b2cac8640 5 monclient: authenticate success, global_id 4522 2024-03-29T10:45:11.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.339+0000 7f0b2cac8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:11.324 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.341+0000 7f0b2559a640 1 -- 172.21.15.84:0/395326532 <== mon.1 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b180017d0 con 0x7f0b2813ee90 2024-03-29T10:45:11.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.341+0000 7f0b2559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:11.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.341+0000 7f0b2559a640 1 -- 172.21.15.84:0/395326532 <== mon.1 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0b180015f0 con 0x7f0b2813ee90 2024-03-29T10:45:11.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.341+0000 7f0b2559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:11.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.341+0000 7f0b2559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:45:11.325 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.341+0000 7f0b2559a640 10 monclient: dump: 2024-03-29T10:45:11.325 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:11.325 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:11.325 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:11.325 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:11.325 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:11.325 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:11.325 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:11.325 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:11.325 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:11.326 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:11.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.341+0000 7f0b2cac8640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:11.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.341+0000 7f0b2cac8640 4 set_mon_vals no callback set 2024-03-29T10:45:11.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.342+0000 7f0b2cac8640 10 monclient: shutdown 2024-03-29T10:45:11.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.342+0000 7f0b2cac8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:11.326 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.342+0000 7f0b2cac8640 1 -- 172.21.15.84:0/395326532 >> v1:172.21.15.86:6789/0 conn(0x7f0b2813ee90 legacy=0x7f0b280adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:11.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.342+0000 7f0b2cac8640 1 -- 172.21.15.84:0/395326532 shutdown_connections 2024-03-29T10:45:11.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.342+0000 7f0b2cac8640 1 -- 172.21.15.84:0/395326532 >> 172.21.15.84:0/395326532 conn(0x7f0b280b35e0 msgr2=0x7f0b280b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:11.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.342+0000 7f0b2cac8640 1 -- 172.21.15.84:0/395326532 shutdown_connections 2024-03-29T10:45:11.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.342+0000 7f0b2cac8640 1 -- 172.21.15.84:0/395326532 wait complete. 2024-03-29T10:45:11.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.343+0000 7f0b2cac8640 10 monclient: build_initial_monmap 2024-03-29T10:45:11.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.343+0000 7f0b2cac8640 10 monclient: monmap: 2024-03-29T10:45:11.327 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:45:11.327 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:11.328 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:45:11.328 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:45:11.328 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:11.328 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:11.328 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:45:11.328 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:45:11.328 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:45:11.328 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:11.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.344+0000 7f0b2cac8640 1 librados: starting msgr at 2024-03-29T10:45:11.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.344+0000 7f0b2cac8640 1 librados: starting objecter 2024-03-29T10:45:11.328 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.344+0000 7f0b2cac8640 1 Processor -- start 2024-03-29T10:45:11.329 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.344+0000 7f0b2cac8640 1 -- start start 2024-03-29T10:45:11.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.344+0000 7f0b2cac8640 1 librados: setting wanted keys 2024-03-29T10:45:11.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.344+0000 7f0b2cac8640 1 librados: calling monclient init 2024-03-29T10:45:11.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.344+0000 7f0b2cac8640 10 monclient: init 2024-03-29T10:45:11.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.345+0000 7f0b2cac8640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:11.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.345+0000 7f0b2cac8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:45:11.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.345+0000 7f0b2cac8640 10 monclient(hunting): picked mon.noname-c con 0x7f0b280af900 addr v1:172.21.15.84:6790/0 2024-03-29T10:45:11.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.345+0000 7f0b2cac8640 10 monclient(hunting): picked mon.noname-a con 0x7f0b280b4100 addr v1:172.21.15.84:6789/0 2024-03-29T10:45:11.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.345+0000 7f0b2cac8640 10 monclient(hunting): picked mon.noname-b con 0x7f0b2813ee90 addr v1:172.21.15.86:6789/0 2024-03-29T10:45:11.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.345+0000 7f0b2cac8640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b28137970 con 0x7f0b280b4100 2024-03-29T10:45:11.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.345+0000 7f0b2cac8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b28148950 con 0x7f0b2813ee90 2024-03-29T10:45:11.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.345+0000 7f0b2cac8640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b28148c30 con 0x7f0b280af900 2024-03-29T10:45:11.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.345+0000 7f0b2cac8640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:11.330 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.345+0000 7f0b2cac8640 10 monclient(hunting): authenticate will time out at 1413.608597s 2024-03-29T10:45:11.331 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.348+0000 7f0b2659c640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f0b280af900 0x7f0b28071df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:33828/0 (socket says 172.21.15.84:33828) 2024-03-29T10:45:11.331 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.348+0000 7f0b2659c640 1 -- 172.21.15.84:0/3830374701 learned_addr learned my addr 172.21.15.84:0/3830374701 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:45:11.331 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.348+0000 7f0b16ffd640 10 client.?.objecter ms_handle_connect 0x7f0b280af900 2024-03-29T10:45:11.331 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.348+0000 7f0b16ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:11.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.348+0000 7f0b16ffd640 1 -- 172.21.15.84:0/3830374701 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2465792118 0 0) 0x7f0b28148c30 con 0x7f0b280af900 2024-03-29T10:45:11.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.348+0000 7f0b16ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:11.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.348+0000 7f0b16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:11.337 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.348+0000 7f0b16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:11.337 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.348+0000 7f0b16ffd640 10 monclient(hunting): my global_id is 4679 2024-03-29T10:45:11.337 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.348+0000 7f0b16ffd640 1 -- 172.21.15.84:0/3830374701 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b08004eb0 con 0x7f0b280af900 2024-03-29T10:45:11.337 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.350+0000 7f0b16ffd640 1 -- 172.21.15.84:0/3830374701 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2138294003 0 0) 0x7f0b08004eb0 con 0x7f0b280af900 2024-03-29T10:45:11.337 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.350+0000 7f0b16ffd640 1 -- 172.21.15.84:0/3830374701 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b28148c30 con 0x7f0b280af900 2024-03-29T10:45:11.337 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.350+0000 7f0b16ffd640 1 -- 172.21.15.84:0/3830374701 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0b1c0019f0 con 0x7f0b280af900 2024-03-29T10:45:11.337 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.350+0000 7f0b16ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:11.337 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.350+0000 7f0b16ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:11.337 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.350+0000 7f0b16ffd640 10 monclient(hunting): dump: 2024-03-29T10:45:11.337 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:11.337 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:11.337 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:11.337 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:11.337 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:11.338 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:11.338 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:11.338 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:11.338 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:11.338 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:11.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.350+0000 7f0b16ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:11.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.350+0000 7f0b16ffd640 10 client.?.objecter ms_handle_connect 0x7f0b280b4100 2024-03-29T10:45:11.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.350+0000 7f0b16ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:11.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.350+0000 7f0b16ffd640 1 -- 172.21.15.84:0/3830374701 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 35762864 0 0) 0x7f0b28137970 con 0x7f0b280b4100 2024-03-29T10:45:11.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.350+0000 7f0b16ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:11.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.350+0000 7f0b16ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:11.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.350+0000 7f0b16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:11.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.350+0000 7f0b16ffd640 10 monclient(hunting): my global_id is 4581 2024-03-29T10:45:11.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.350+0000 7f0b16ffd640 1 -- 172.21.15.84:0/3830374701 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b080056d0 con 0x7f0b280b4100 2024-03-29T10:45:11.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.351+0000 7f0b16ffd640 1 -- 172.21.15.84:0/3830374701 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 9032000 0 0) 0x7f0b28148c30 con 0x7f0b280af900 2024-03-29T10:45:11.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.351+0000 7f0b16ffd640 1 -- 172.21.15.84:0/3830374701 >> v1:172.21.15.86:6789/0 conn(0x7f0b2813ee90 legacy=0x7f0b28137260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:11.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.351+0000 7f0b16ffd640 1 -- 172.21.15.84:0/3830374701 >> v1:172.21.15.84:6789/0 conn(0x7f0b280b4100 legacy=0x7f0b28136b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:11.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.351+0000 7f0b16ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:45:11.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.351+0000 7f0b16ffd640 1 monclient: found mon.c 2024-03-29T10:45:11.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.351+0000 7f0b16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:11.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.351+0000 7f0b16ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:45:11.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.351+0000 7f0b16ffd640 1 -- 172.21.15.84:0/3830374701 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b28072500 con 0x7f0b280af900 2024-03-29T10:45:11.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.351+0000 7f0b16ffd640 10 monclient: _finish_auth 0 2024-03-29T10:45:11.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.351+0000 7f0b16ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:45:11.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.351+0000 7f0b16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:11.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.351+0000 7f0b2cac8640 5 monclient: authenticate success, global_id 4679 2024-03-29T10:45:11.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.351+0000 7f0b2cac8640 10 monclient: _renew_subs 2024-03-29T10:45:11.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.351+0000 7f0b2cac8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:45:11.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.351+0000 7f0b2cac8640 1 -- 172.21.15.84:0/3830374701 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b28151b10 con 0x7f0b280af900 2024-03-29T10:45:11.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.351+0000 7f0b2cac8640 10 client.4679.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:11.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.351+0000 7f0b2cac8640 10 monclient: _renew_subs 2024-03-29T10:45:11.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.351+0000 7f0b2cac8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:45:11.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.351+0000 7f0b2cac8640 1 -- 172.21.15.84:0/3830374701 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b28149410 con 0x7f0b280af900 2024-03-29T10:45:11.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.351+0000 7f0b2cac8640 1 librados: init done 2024-03-29T10:45:11.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.352+0000 7f0b16ffd640 1 -- 172.21.15.84:0/3830374701 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b1c001d10 con 0x7f0b280af900 2024-03-29T10:45:11.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.352+0000 7f0b16ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:11.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.353+0000 7f0b2cac8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:11.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.353+0000 7f0b2cac8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:11.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.353+0000 7f0b2cac8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:45:11.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.353+0000 7f0b2cac8640 1 -- 172.21.15.84:0/3830374701 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b28149710 con 0x7f0b280af900 2024-03-29T10:45:11.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.353+0000 7f0b16ffd640 1 -- 172.21.15.84:0/3830374701 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0b1c0019f0 con 0x7f0b280af900 2024-03-29T10:45:11.341 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.353+0000 7f0b16ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:11.341 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.353+0000 7f0b16ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:11.341 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.353+0000 7f0b16ffd640 10 monclient: dump: 2024-03-29T10:45:11.341 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:11.341 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:11.341 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:11.341 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:11.341 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:11.341 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:11.341 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:11.341 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:11.341 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:11.341 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:11.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.354+0000 7f0b24d99640 4 set_mon_vals no callback set 2024-03-29T10:45:11.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.355+0000 7f0b16ffd640 1 -- 172.21.15.84:0/3830374701 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f0b1c003920 con 0x7f0b280af900 2024-03-29T10:45:11.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.356+0000 7f0b16ffd640 1 -- 172.21.15.84:0/3830374701 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1523682299 0 0) 0x7f0b1c084670 con 0x7f0b280af900 2024-03-29T10:45:11.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.356+0000 7f0b16ffd640 10 client.4679.objecter ms_dispatch 0x7f0b28000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:45:11.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.356+0000 7f0b16ffd640 3 client.4679.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:45:11.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.356+0000 7f0b16ffd640 3 client.4679.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:45:11.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.356+0000 7f0b16ffd640 20 client.4679.objecter dump_active .. 0 homeless 2024-03-29T10:45:11.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.356+0000 7f0b25d9b640 10 monclient: get_auth_request con 0x7f0b0806e000 auth_method 0 2024-03-29T10:45:11.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.356+0000 7f0b16ffd640 10 client.4679.objecter ms_handle_connect 0x7f0b0806e000 2024-03-29T10:45:11.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.361+0000 7f0b16ffd640 1 -- 172.21.15.84:0/3830374701 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b1c04fd80 con 0x7f0b280af900 2024-03-29T10:45:11.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.361+0000 7f0b16ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:11.344 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.361+0000 7f0b16ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:11.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.682+0000 7f0b2cac8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:11.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.682+0000 7f0b2cac8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:11.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.682+0000 7f0b2cac8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:45:11.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.682+0000 7f0b2cac8640 1 -- 172.21.15.84:0/3830374701 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b28149710 con 0x7f0b280af900 2024-03-29T10:45:11.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.683+0000 7f0b16ffd640 1 -- 172.21.15.84:0/3830374701 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 879675878) 0x7f0b1c001850 con 0x7f0b280af900 2024-03-29T10:45:11.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.683+0000 7f0b16ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:11.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.683+0000 7f0b16ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:11.667 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:45:11.667 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":11,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:44:56.255219+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:56.255219+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:11.669 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.687+0000 7f0b2cac8640 10 librados: watch_flush enter 2024-03-29T10:45:11.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.688+0000 7f0b2cac8640 10 librados: watch_flush exit 2024-03-29T10:45:11.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.688+0000 7f0b2cac8640 20 client.4679.objecter shutdown clearing up homeless session... 2024-03-29T10:45:11.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.688+0000 7f0b2cac8640 10 client.4679.objecter successfully canceled tick 2024-03-29T10:45:11.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.688+0000 7f0b2cac8640 1 -- 172.21.15.84:0/3830374701 >> v1:172.21.15.86:6816/3337225337 conn(0x7f0b0806e000 legacy=0x7f0b080704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:11.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.688+0000 7f0b2cac8640 10 monclient: shutdown 2024-03-29T10:45:11.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.688+0000 7f0b2cac8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:11.671 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.688+0000 7f0b2cac8640 1 -- 172.21.15.84:0/3830374701 >> v1:172.21.15.84:6790/0 conn(0x7f0b280af900 legacy=0x7f0b28071df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:11.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.689+0000 7f0b2cac8640 1 -- 172.21.15.84:0/3830374701 shutdown_connections 2024-03-29T10:45:11.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.689+0000 7f0b2cac8640 1 -- 172.21.15.84:0/3830374701 >> 172.21.15.84:0/3830374701 conn(0x7f0b280b35e0 msgr2=0x7f0b280ad280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:11.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.689+0000 7f0b2cac8640 1 -- 172.21.15.84:0/3830374701 shutdown_connections 2024-03-29T10:45:11.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.689+0000 7f0b2cac8640 1 -- 172.21.15.84:0/3830374701 wait complete. 2024-03-29T10:45:11.672 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:11.690+0000 7f0b2cac8640 1 librados: shutdown 2024-03-29T10:45:11.692 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:18.499 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:18.640 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.657+0000 7fa671eba640 10 monclient: get_monmap_and_config 2024-03-29T10:45:18.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.658+0000 7fa671eba640 10 monclient: build_initial_monmap 2024-03-29T10:45:18.641 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.658+0000 7fa671eba640 10 monclient: monmap: 2024-03-29T10:45:18.641 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:45:18.641 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:18.641 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:45:18.658979+0000 2024-03-29T10:45:18.641 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:45:18.658979+0000 2024-03-29T10:45:18.641 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:18.641 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:18.641 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:45:18.641 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:45:18.641 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:45:18.641 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:18.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.660+0000 7fa671eba640 1 Processor -- start 2024-03-29T10:45:18.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.660+0000 7fa671eba640 1 -- start start 2024-03-29T10:45:18.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.660+0000 7fa671eba640 10 monclient: init 2024-03-29T10:45:18.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.661+0000 7fa671eba640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:18.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.661+0000 7fa671eba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:45:18.644 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.661+0000 7fa671eba640 10 monclient(hunting): picked mon.noname-b con 0x7fa66c0b0770 addr v1:172.21.15.84:6790/0 2024-03-29T10:45:18.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.663+0000 7fa671eba640 10 monclient(hunting): picked mon.noname-a con 0x7fa66c14a700 addr v1:172.21.15.84:6789/0 2024-03-29T10:45:18.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.663+0000 7fa671eba640 10 monclient(hunting): picked mon.noname-c con 0x7fa66c0ac5a0 addr v1:172.21.15.86:6789/0 2024-03-29T10:45:18.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.663+0000 7fa671eba640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa66c0aba70 con 0x7fa66c14a700 2024-03-29T10:45:18.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.663+0000 7fa671eba640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa66c0aecb0 con 0x7fa66c0ac5a0 2024-03-29T10:45:18.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.663+0000 7fa671eba640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa66c0af150 con 0x7fa66c0b0770 2024-03-29T10:45:18.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.663+0000 7fa671eba640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:18.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.663+0000 7fa671eba640 10 monclient(hunting): authenticate will time out at 1420.926500s 2024-03-29T10:45:18.650 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.665+0000 7fa66affd640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fa66c14a700 0x7fa66c0abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:46031/0 (socket says 172.21.15.84:46031) 2024-03-29T10:45:18.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.665+0000 7fa66affd640 1 -- 172.21.15.84:0/994127112 learned_addr learned my addr 172.21.15.84:0/994127112 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:45:18.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.665+0000 7fa66a7fc640 1 -- 172.21.15.84:0/994127112 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3589125764 0 0) 0x7fa66c0aba70 con 0x7fa66c14a700 2024-03-29T10:45:18.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.665+0000 7fa66a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:18.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.665+0000 7fa66a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:18.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.665+0000 7fa66a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:18.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.665+0000 7fa66a7fc640 10 monclient(hunting): my global_id is 4584 2024-03-29T10:45:18.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.666+0000 7fa66a7fc640 1 -- 172.21.15.84:0/994127112 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa654003520 con 0x7fa66c14a700 2024-03-29T10:45:18.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.666+0000 7fa66a7fc640 1 -- 172.21.15.84:0/994127112 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3283167027 0 0) 0x7fa654003520 con 0x7fa66c14a700 2024-03-29T10:45:18.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.666+0000 7fa66a7fc640 1 -- 172.21.15.84:0/994127112 >> v1:172.21.15.84:6790/0 conn(0x7fa66c0b0770 legacy=0x7fa66c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:18.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.666+0000 7fa66a7fc640 1 -- 172.21.15.84:0/994127112 >> v1:172.21.15.86:6789/0 conn(0x7fa66c0ac5a0 legacy=0x7fa66c151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:18.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.666+0000 7fa66a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:45:18.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.666+0000 7fa66a7fc640 1 monclient: found mon.noname-a 2024-03-29T10:45:18.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.666+0000 7fa66a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:18.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.666+0000 7fa66a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T10:45:18.651 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.666+0000 7fa66a7fc640 1 -- 172.21.15.84:0/994127112 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa66c0ac9d0 con 0x7fa66c14a700 2024-03-29T10:45:18.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.666+0000 7fa66a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:45:18.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.666+0000 7fa66a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:45:18.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.666+0000 7fa66a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:18.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.666+0000 7fa66a7fc640 1 -- 172.21.15.84:0/994127112 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa660001690 con 0x7fa66c14a700 2024-03-29T10:45:18.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.666+0000 7fa66a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:18.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.666+0000 7fa66a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:18.652 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.666+0000 7fa66a7fc640 10 monclient: dump: 2024-03-29T10:45:18.652 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:18.652 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:18.652 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:18.652 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:18.652 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:18.652 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:18.652 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:18.652 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:18.652 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:18.653 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:18.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.666+0000 7fa671eba640 5 monclient: authenticate success, global_id 4584 2024-03-29T10:45:18.653 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.666+0000 7fa671eba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:18.654 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.667+0000 7fa66a7fc640 1 -- 172.21.15.84:0/994127112 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa660001860 con 0x7fa66c14a700 2024-03-29T10:45:18.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.667+0000 7fa66a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:18.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.667+0000 7fa66a7fc640 1 -- 172.21.15.84:0/994127112 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa660001cb0 con 0x7fa66c14a700 2024-03-29T10:45:18.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.667+0000 7fa66a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:18.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.667+0000 7fa66a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:18.655 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.667+0000 7fa66a7fc640 10 monclient: dump: 2024-03-29T10:45:18.655 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:18.655 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:18.655 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:18.655 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:18.655 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:18.655 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:18.655 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:18.655 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:18.655 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:18.656 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:18.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.667+0000 7fa671eba640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:18.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.667+0000 7fa671eba640 4 set_mon_vals no callback set 2024-03-29T10:45:18.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.668+0000 7fa671eba640 10 monclient: shutdown 2024-03-29T10:45:18.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.668+0000 7fa671eba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:18.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.668+0000 7fa671eba640 1 -- 172.21.15.84:0/994127112 >> v1:172.21.15.84:6789/0 conn(0x7fa66c14a700 legacy=0x7fa66c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:18.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.668+0000 7fa671eba640 1 -- 172.21.15.84:0/994127112 shutdown_connections 2024-03-29T10:45:18.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.668+0000 7fa671eba640 1 -- 172.21.15.84:0/994127112 >> 172.21.15.84:0/994127112 conn(0x7fa66c06b660 msgr2=0x7fa66c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:18.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.670+0000 7fa671eba640 1 -- 172.21.15.84:0/994127112 shutdown_connections 2024-03-29T10:45:18.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.670+0000 7fa671eba640 1 -- 172.21.15.84:0/994127112 wait complete. 2024-03-29T10:45:18.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.671+0000 7fa671eba640 10 monclient: build_initial_monmap 2024-03-29T10:45:18.656 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.671+0000 7fa671eba640 10 monclient: monmap: 2024-03-29T10:45:18.656 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:45:18.656 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:18.656 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:45:18.657 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:45:18.657 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:18.657 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:18.657 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:45:18.657 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:45:18.657 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:45:18.657 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:18.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.671+0000 7fa671eba640 1 librados: starting msgr at 2024-03-29T10:45:18.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.671+0000 7fa671eba640 1 librados: starting objecter 2024-03-29T10:45:18.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.671+0000 7fa671eba640 1 Processor -- start 2024-03-29T10:45:18.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.672+0000 7fa671eba640 1 -- start start 2024-03-29T10:45:18.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.672+0000 7fa671eba640 1 librados: setting wanted keys 2024-03-29T10:45:18.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.672+0000 7fa671eba640 1 librados: calling monclient init 2024-03-29T10:45:18.657 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.672+0000 7fa671eba640 10 monclient: init 2024-03-29T10:45:18.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.672+0000 7fa671eba640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:18.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.672+0000 7fa671eba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:45:18.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.675+0000 7fa671eba640 10 monclient(hunting): picked mon.noname-a con 0x7fa66c0ac5a0 addr v1:172.21.15.84:6789/0 2024-03-29T10:45:18.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.675+0000 7fa671eba640 10 monclient(hunting): picked mon.noname-b con 0x7fa66c0b0770 addr v1:172.21.15.86:6789/0 2024-03-29T10:45:18.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.675+0000 7fa66b7fe640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7fa66c0ac5a0 0x7fa66c1471e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:46225/0 (socket says 172.21.15.84:46225) 2024-03-29T10:45:18.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.675+0000 7fa671eba640 10 monclient(hunting): picked mon.noname-c con 0x7fa66c14a700 addr v1:172.21.15.84:6790/0 2024-03-29T10:45:18.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.675+0000 7fa66b7fe640 1 -- 172.21.15.84:0/507533133 learned_addr learned my addr 172.21.15.84:0/507533133 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:45:18.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.675+0000 7fa671eba640 1 -- 172.21.15.84:0/507533133 --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa66c12a2d0 con 0x7fa66c0ac5a0 2024-03-29T10:45:18.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.675+0000 7fa653fff640 10 client.?.objecter ms_handle_connect 0x7fa66c0ac5a0 2024-03-29T10:45:18.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.675+0000 7fa653fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:18.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.675+0000 7fa653fff640 1 -- 172.21.15.84:0/507533133 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860101289 0 0) 0x7fa66c12a2d0 con 0x7fa66c0ac5a0 2024-03-29T10:45:18.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.675+0000 7fa671eba640 1 -- 172.21.15.84:0/507533133 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa66c12a5b0 con 0x7fa66c0b0770 2024-03-29T10:45:18.660 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.675+0000 7fa671eba640 1 -- 172.21.15.84:0/507533133 --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa66c1281c0 con 0x7fa66c14a700 2024-03-29T10:45:18.661 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.675+0000 7fa671eba640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:18.661 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.675+0000 7fa671eba640 10 monclient(hunting): authenticate will time out at 1420.939084s 2024-03-29T10:45:18.661 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.676+0000 7fa653fff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:18.661 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.676+0000 7fa653fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:18.661 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.676+0000 7fa653fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:18.661 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.676+0000 7fa653fff640 10 monclient(hunting): my global_id is 4587 2024-03-29T10:45:18.661 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.676+0000 7fa653fff640 1 -- 172.21.15.84:0/507533133 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa658004eb0 con 0x7fa66c0ac5a0 2024-03-29T10:45:18.661 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.676+0000 7fa653fff640 10 client.?.objecter ms_handle_connect 0x7fa66c14a700 2024-03-29T10:45:18.661 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.676+0000 7fa653fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:18.663 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa653fff640 1 -- 172.21.15.84:0/507533133 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 176713265 0 0) 0x7fa658004eb0 con 0x7fa66c0ac5a0 2024-03-29T10:45:18.663 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa653fff640 1 -- 172.21.15.84:0/507533133 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa66c12a2d0 con 0x7fa66c0ac5a0 2024-03-29T10:45:18.663 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa653fff640 1 -- 172.21.15.84:0/507533133 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2120974633 0 0) 0x7fa66c1281c0 con 0x7fa66c14a700 2024-03-29T10:45:18.663 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa653fff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:18.663 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa653fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:18.663 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa653fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:18.663 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa653fff640 10 monclient(hunting): my global_id is 4685 2024-03-29T10:45:18.663 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa653fff640 1 -- 172.21.15.84:0/507533133 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa658004eb0 con 0x7fa66c14a700 2024-03-29T10:45:18.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa653fff640 1 -- 172.21.15.84:0/507533133 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa65c001db0 con 0x7fa66c0ac5a0 2024-03-29T10:45:18.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa653fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:18.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa653fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:18.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa653fff640 10 monclient(hunting): dump: 2024-03-29T10:45:18.664 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:18.664 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:18.664 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:18.664 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:18.664 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:18.664 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:18.664 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:18.664 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:18.664 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:18.664 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:18.664 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa653fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:18.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa653fff640 10 client.?.objecter ms_handle_connect 0x7fa66c0b0770 2024-03-29T10:45:18.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa653fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:18.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa653fff640 1 -- 172.21.15.84:0/507533133 <== mon.0 v1:172.21.15.84:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1624039262 0 0) 0x7fa66c12a2d0 con 0x7fa66c0ac5a0 2024-03-29T10:45:18.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa653fff640 1 -- 172.21.15.84:0/507533133 >> v1:172.21.15.84:6790/0 conn(0x7fa66c14a700 legacy=0x7fa66c1e6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:18.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa653fff640 1 -- 172.21.15.84:0/507533133 >> v1:172.21.15.86:6789/0 conn(0x7fa66c0b0770 legacy=0x7fa66c1db980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:18.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa653fff640 10 monclient: _finish_hunting 0 2024-03-29T10:45:18.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa653fff640 1 monclient: found mon.a 2024-03-29T10:45:18.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa653fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:18.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa653fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:45:18.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa653fff640 1 -- 172.21.15.84:0/507533133 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa66c128350 con 0x7fa66c0ac5a0 2024-03-29T10:45:18.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa653fff640 10 monclient: _finish_auth 0 2024-03-29T10:45:18.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa653fff640 10 monclient: _check_auth_tickets 2024-03-29T10:45:18.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa653fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:18.665 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa671eba640 5 monclient: authenticate success, global_id 4587 2024-03-29T10:45:18.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa671eba640 10 monclient: _renew_subs 2024-03-29T10:45:18.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa671eba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:45:18.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa671eba640 1 -- 172.21.15.84:0/507533133 --> v1:172.21.15.84:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa66c1478f0 con 0x7fa66c0ac5a0 2024-03-29T10:45:18.666 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa671eba640 10 client.4587.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:18.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa671eba640 10 monclient: _renew_subs 2024-03-29T10:45:18.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa671eba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:45:18.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa671eba640 1 -- 172.21.15.84:0/507533133 --> v1:172.21.15.84:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa66c1dc090 con 0x7fa66c0ac5a0 2024-03-29T10:45:18.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.679+0000 7fa671eba640 1 librados: init done 2024-03-29T10:45:18.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.682+0000 7fa653fff640 1 -- 172.21.15.84:0/507533133 <== mon.0 v1:172.21.15.84:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa65c001f10 con 0x7fa66c0ac5a0 2024-03-29T10:45:18.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.682+0000 7fa653fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:18.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.682+0000 7fa653fff640 1 -- 172.21.15.84:0/507533133 <== mon.0 v1:172.21.15.84:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fa65c003de0 con 0x7fa66c0ac5a0 2024-03-29T10:45:18.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.682+0000 7fa653fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:18.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.682+0000 7fa653fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:18.667 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.682+0000 7fa653fff640 10 monclient: dump: 2024-03-29T10:45:18.667 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:18.667 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:18.667 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:18.668 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:18.668 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:18.668 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:18.668 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:18.668 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:18.668 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:18.668 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:18.668 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.682+0000 7fa653fff640 1 -- 172.21.15.84:0/507533133 <== mon.0 v1:172.21.15.84:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7fa65c004000 con 0x7fa66c0ac5a0 2024-03-29T10:45:18.668 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.682+0000 7fa653fff640 1 -- 172.21.15.84:0/507533133 <== mon.0 v1:172.21.15.84:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1523682299 0 0) 0x7fa65c084ff0 con 0x7fa66c0ac5a0 2024-03-29T10:45:18.668 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.682+0000 7fa653fff640 10 client.4587.objecter ms_dispatch 0x7fa66c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:45:18.668 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.682+0000 7fa653fff640 3 client.4587.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:45:18.668 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.682+0000 7fa653fff640 3 client.4587.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:45:18.668 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.682+0000 7fa653fff640 20 client.4587.objecter dump_active .. 0 homeless 2024-03-29T10:45:18.668 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.682+0000 7fa669ffb640 4 set_mon_vals no callback set 2024-03-29T10:45:18.668 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.683+0000 7fa671eba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:18.669 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.683+0000 7fa671eba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:18.669 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.683+0000 7fa671eba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:45:18.669 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.683+0000 7fa671eba640 1 -- 172.21.15.84:0/507533133 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa64c005080 con 0x7fa66c0ac5a0 2024-03-29T10:45:18.669 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.684+0000 7fa66affd640 10 monclient: get_auth_request con 0x7fa65806dd30 auth_method 0 2024-03-29T10:45:18.669 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.684+0000 7fa653fff640 10 client.4587.objecter ms_handle_connect 0x7fa65806dd30 2024-03-29T10:45:18.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.691+0000 7fa653fff640 1 -- 172.21.15.84:0/507533133 <== mon.0 v1:172.21.15.84:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa65c050950 con 0x7fa66c0ac5a0 2024-03-29T10:45:18.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.691+0000 7fa653fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:18.674 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:18.691+0000 7fa653fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:19.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:19.048+0000 7fa671eba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:19.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:19.048+0000 7fa671eba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:19.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:19.048+0000 7fa671eba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.84:6789/0 2024-03-29T10:45:19.031 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:19.048+0000 7fa671eba640 1 -- 172.21.15.84:0/507533133 --> v1:172.21.15.84:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa64c005330 con 0x7fa66c0ac5a0 2024-03-29T10:45:19.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:19.050+0000 7fa653fff640 1 -- 172.21.15.84:0/507533133 <== mon.0 v1:172.21.15.84:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 879675878) 0x7fa65c052880 con 0x7fa66c0ac5a0 2024-03-29T10:45:19.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:19.050+0000 7fa653fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:19.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:19.050+0000 7fa653fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:19.034 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:45:19.034 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":11,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:44:56.255219+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:56.255219+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:19.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:19.055+0000 7fa671eba640 10 librados: watch_flush enter 2024-03-29T10:45:19.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:19.055+0000 7fa671eba640 10 librados: watch_flush exit 2024-03-29T10:45:19.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:19.055+0000 7fa671eba640 20 client.4587.objecter shutdown clearing up homeless session... 2024-03-29T10:45:19.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:19.055+0000 7fa671eba640 10 client.4587.objecter successfully canceled tick 2024-03-29T10:45:19.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:19.056+0000 7fa671eba640 1 -- 172.21.15.84:0/507533133 >> v1:172.21.15.86:6816/3337225337 conn(0x7fa65806dd30 legacy=0x7fa6580701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:19.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:19.056+0000 7fa671eba640 10 monclient: shutdown 2024-03-29T10:45:19.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:19.056+0000 7fa671eba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:19.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:19.056+0000 7fa671eba640 1 -- 172.21.15.84:0/507533133 >> v1:172.21.15.84:6789/0 conn(0x7fa66c0ac5a0 legacy=0x7fa66c1471e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:19.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:19.056+0000 7fa671eba640 1 -- 172.21.15.84:0/507533133 shutdown_connections 2024-03-29T10:45:19.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:19.056+0000 7fa671eba640 1 -- 172.21.15.84:0/507533133 >> 172.21.15.84:0/507533133 conn(0x7fa66c06b660 msgr2=0x7fa66c0abe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:19.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:19.057+0000 7fa671eba640 1 -- 172.21.15.84:0/507533133 shutdown_connections 2024-03-29T10:45:19.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:19.058+0000 7fa671eba640 1 -- 172.21.15.84:0/507533133 wait complete. 2024-03-29T10:45:19.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:19.058+0000 7fa671eba640 1 librados: shutdown 2024-03-29T10:45:19.064 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:25.871 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:26.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.032+0000 7f0ba663c640 10 monclient: get_monmap_and_config 2024-03-29T10:45:26.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.032+0000 7f0ba663c640 10 monclient: build_initial_monmap 2024-03-29T10:45:26.015 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.032+0000 7f0ba663c640 10 monclient: monmap: 2024-03-29T10:45:26.016 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:45:26.016 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:26.016 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:45:26.033826+0000 2024-03-29T10:45:26.016 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:45:26.033826+0000 2024-03-29T10:45:26.016 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:26.016 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:26.016 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:45:26.016 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:45:26.016 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:45:26.016 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:26.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.033+0000 7f0ba663c640 1 Processor -- start 2024-03-29T10:45:26.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.033+0000 7f0ba663c640 1 -- start start 2024-03-29T10:45:26.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.033+0000 7f0ba663c640 10 monclient: init 2024-03-29T10:45:26.017 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.035+0000 7f0ba663c640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:26.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.035+0000 7f0ba663c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:45:26.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.035+0000 7f0ba663c640 10 monclient(hunting): picked mon.noname-a con 0x7f0ba01457f0 addr v1:172.21.15.84:6789/0 2024-03-29T10:45:26.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.035+0000 7f0ba663c640 10 monclient(hunting): picked mon.noname-b con 0x7f0ba013f650 addr v1:172.21.15.84:6790/0 2024-03-29T10:45:26.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.035+0000 7f0ba663c640 10 monclient(hunting): picked mon.noname-c con 0x7f0ba00a9d70 addr v1:172.21.15.86:6789/0 2024-03-29T10:45:26.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.035+0000 7f0ba663c640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba00aaef0 con 0x7f0ba01457f0 2024-03-29T10:45:26.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.035+0000 7f0ba663c640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba0145bd0 con 0x7f0ba00a9d70 2024-03-29T10:45:26.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.035+0000 7f0ba663c640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba0145eb0 con 0x7f0ba013f650 2024-03-29T10:45:26.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.035+0000 7f0ba663c640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:26.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.035+0000 7f0ba663c640 10 monclient(hunting): authenticate will time out at 1428.298284s 2024-03-29T10:45:26.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.035+0000 7f0b9f7fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f0ba013f650 0x7f0ba00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:47684/0 (socket says 172.21.15.84:47684) 2024-03-29T10:45:26.018 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.035+0000 7f0b9f7fe640 1 -- 172.21.15.84:0/3995568686 learned_addr learned my addr 172.21.15.84:0/3995568686 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:45:26.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.035+0000 7f0b9effd640 1 -- 172.21.15.84:0/3995568686 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021223434 0 0) 0x7f0ba0145eb0 con 0x7f0ba013f650 2024-03-29T10:45:26.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.035+0000 7f0b9effd640 10 monclient(hunting): _negotiate 2024-03-29T10:45:26.019 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.035+0000 7f0b9effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:26.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.035+0000 7f0b9effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:26.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.035+0000 7f0b9effd640 10 monclient(hunting): my global_id is 4688 2024-03-29T10:45:26.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.035+0000 7f0b9effd640 1 -- 172.21.15.84:0/3995568686 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b8c003520 con 0x7f0ba013f650 2024-03-29T10:45:26.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.036+0000 7f0b9effd640 1 -- 172.21.15.84:0/3995568686 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1171081362 0 0) 0x7f0b8c003520 con 0x7f0ba013f650 2024-03-29T10:45:26.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.036+0000 7f0b9effd640 1 -- 172.21.15.84:0/3995568686 >> v1:172.21.15.86:6789/0 conn(0x7f0ba00a9d70 legacy=0x7f0ba014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:45:26.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.036+0000 7f0b9effd640 1 -- 172.21.15.84:0/3995568686 >> v1:172.21.15.84:6789/0 conn(0x7f0ba01457f0 legacy=0x7f0ba013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:26.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.036+0000 7f0b9effd640 10 monclient: _finish_hunting 0 2024-03-29T10:45:26.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.036+0000 7f0b9effd640 1 monclient: found mon.noname-b 2024-03-29T10:45:26.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.036+0000 7f0b9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:26.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.036+0000 7f0b9effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:45:26.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.036+0000 7f0b9effd640 1 -- 172.21.15.84:0/3995568686 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba0146340 con 0x7f0ba013f650 2024-03-29T10:45:26.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.036+0000 7f0b9effd640 10 monclient: _finish_auth 0 2024-03-29T10:45:26.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.036+0000 7f0b9effd640 10 monclient: _check_auth_tickets 2024-03-29T10:45:26.020 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.036+0000 7f0b9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:26.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.036+0000 7f0b9effd640 1 -- 172.21.15.84:0/3995568686 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0b90001690 con 0x7f0ba013f650 2024-03-29T10:45:26.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.036+0000 7f0ba663c640 5 monclient: authenticate success, global_id 4688 2024-03-29T10:45:26.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.036+0000 7f0ba663c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:26.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.036+0000 7f0b9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:26.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.036+0000 7f0b9effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:26.021 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.036+0000 7f0b9effd640 10 monclient: dump: 2024-03-29T10:45:26.021 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:26.021 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:26.021 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:26.021 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:26.021 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:26.021 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:26.021 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:26.021 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:26.022 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:26.022 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:26.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.036+0000 7f0ba663c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:26.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.037+0000 7f0b9effd640 1 -- 172.21.15.84:0/3995568686 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b90001860 con 0x7f0ba013f650 2024-03-29T10:45:26.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.037+0000 7f0b9effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:26.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.037+0000 7f0ba663c640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:26.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.037+0000 7f0ba663c640 4 set_mon_vals no callback set 2024-03-29T10:45:26.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.037+0000 7f0b9effd640 1 -- 172.21.15.84:0/3995568686 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0b90001cb0 con 0x7f0ba013f650 2024-03-29T10:45:26.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.037+0000 7f0b9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:26.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.037+0000 7f0b9effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:26.022 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.037+0000 7f0b9effd640 10 monclient: dump: 2024-03-29T10:45:26.022 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:26.022 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:26.022 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:26.022 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:26.023 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:26.023 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:26.023 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:26.023 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:26.023 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:26.023 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:26.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.038+0000 7f0ba663c640 10 monclient: shutdown 2024-03-29T10:45:26.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.038+0000 7f0ba663c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:26.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.038+0000 7f0ba663c640 1 -- 172.21.15.84:0/3995568686 >> v1:172.21.15.84:6790/0 conn(0x7f0ba013f650 legacy=0x7f0ba00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:26.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.038+0000 7f0ba663c640 1 -- 172.21.15.84:0/3995568686 shutdown_connections 2024-03-29T10:45:26.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.038+0000 7f0ba663c640 1 -- 172.21.15.84:0/3995568686 >> 172.21.15.84:0/3995568686 conn(0x7f0ba0066a00 msgr2=0x7f0ba0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:26.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.038+0000 7f0ba663c640 1 -- 172.21.15.84:0/3995568686 shutdown_connections 2024-03-29T10:45:26.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.038+0000 7f0ba663c640 1 -- 172.21.15.84:0/3995568686 wait complete. 2024-03-29T10:45:26.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.039+0000 7f0ba663c640 10 monclient: build_initial_monmap 2024-03-29T10:45:26.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.039+0000 7f0ba663c640 10 monclient: monmap: 2024-03-29T10:45:26.037 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:45:26.037 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:26.038 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:45:26.038 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:45:26.038 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:26.038 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:26.038 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:45:26.038 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:45:26.038 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:45:26.039 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:26.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.039+0000 7f0ba663c640 1 librados: starting msgr at 2024-03-29T10:45:26.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.039+0000 7f0ba663c640 1 librados: starting objecter 2024-03-29T10:45:26.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.039+0000 7f0ba663c640 1 Processor -- start 2024-03-29T10:45:26.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.040+0000 7f0ba663c640 1 -- start start 2024-03-29T10:45:26.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.040+0000 7f0ba663c640 1 librados: setting wanted keys 2024-03-29T10:45:26.039 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.040+0000 7f0ba663c640 1 librados: calling monclient init 2024-03-29T10:45:26.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.040+0000 7f0ba663c640 10 monclient: init 2024-03-29T10:45:26.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.040+0000 7f0ba663c640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:26.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.041+0000 7f0ba663c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:45:26.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.041+0000 7f0ba663c640 10 monclient(hunting): picked mon.noname-c con 0x7f0ba00a9d70 addr v1:172.21.15.84:6790/0 2024-03-29T10:45:26.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.041+0000 7f0ba663c640 10 monclient(hunting): picked mon.noname-a con 0x7f0ba013f650 addr v1:172.21.15.84:6789/0 2024-03-29T10:45:26.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.041+0000 7f0ba663c640 10 monclient(hunting): picked mon.noname-b con 0x7f0ba01457f0 addr v1:172.21.15.86:6789/0 2024-03-29T10:45:26.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.041+0000 7f0ba663c640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba0144620 con 0x7f0ba013f650 2024-03-29T10:45:26.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.041+0000 7f0ba663c640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba0144340 con 0x7f0ba01457f0 2024-03-29T10:45:26.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.041+0000 7f0ba663c640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba0145230 con 0x7f0ba00a9d70 2024-03-29T10:45:26.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.041+0000 7f0ba663c640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:26.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.041+0000 7f0ba663c640 10 monclient(hunting): authenticate will time out at 1428.304197s 2024-03-29T10:45:26.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.041+0000 7f0ba4bd9640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f0ba01457f0 0x7f0ba01eeaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:48006/0 (socket says 172.21.15.84:48006) 2024-03-29T10:45:26.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.041+0000 7f0ba4bd9640 1 -- 172.21.15.84:0/2679480616 learned_addr learned my addr 172.21.15.84:0/2679480616 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:45:26.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.041+0000 7f0b9cff9640 10 client.?.objecter ms_handle_connect 0x7f0ba01457f0 2024-03-29T10:45:26.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.041+0000 7f0b9cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:26.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.041+0000 7f0b9cff9640 10 client.?.objecter ms_handle_connect 0x7f0ba00a9d70 2024-03-29T10:45:26.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.041+0000 7f0b9cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:26.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.041+0000 7f0b9cff9640 1 -- 172.21.15.84:0/2679480616 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4268299870 0 0) 0x7f0ba0144340 con 0x7f0ba01457f0 2024-03-29T10:45:26.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.041+0000 7f0b9cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:45:26.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.041+0000 7f0b9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:26.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.041+0000 7f0b9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:26.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.041+0000 7f0b9cff9640 10 monclient(hunting): my global_id is 4528 2024-03-29T10:45:26.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.041+0000 7f0b9cff9640 1 -- 172.21.15.84:0/2679480616 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b80004eb0 con 0x7f0ba01457f0 2024-03-29T10:45:26.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.042+0000 7f0b9cff9640 10 client.?.objecter ms_handle_connect 0x7f0ba013f650 2024-03-29T10:45:26.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.042+0000 7f0b9cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:26.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.042+0000 7f0b9cff9640 1 -- 172.21.15.84:0/2679480616 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1586001358 0 0) 0x7f0ba0145230 con 0x7f0ba00a9d70 2024-03-29T10:45:26.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.042+0000 7f0b9cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:45:26.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.042+0000 7f0b9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:26.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.042+0000 7f0b9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:26.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.042+0000 7f0b9cff9640 10 monclient(hunting): my global_id is 4691 2024-03-29T10:45:26.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.042+0000 7f0b9cff9640 1 -- 172.21.15.84:0/2679480616 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ba0144340 con 0x7f0ba00a9d70 2024-03-29T10:45:26.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.042+0000 7f0b9cff9640 1 -- 172.21.15.84:0/2679480616 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199486807 0 0) 0x7f0ba0144620 con 0x7f0ba013f650 2024-03-29T10:45:26.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.042+0000 7f0b9cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:45:26.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.042+0000 7f0b9cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:26.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.042+0000 7f0b9cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:26.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.042+0000 7f0b9cff9640 10 monclient(hunting): my global_id is 4593 2024-03-29T10:45:26.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.042+0000 7f0b9cff9640 1 -- 172.21.15.84:0/2679480616 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ba0145230 con 0x7f0ba013f650 2024-03-29T10:45:26.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.042+0000 7f0b9cff9640 1 -- 172.21.15.84:0/2679480616 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2421016526 0 0) 0x7f0b80004eb0 con 0x7f0ba01457f0 2024-03-29T10:45:26.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.042+0000 7f0b9cff9640 1 -- 172.21.15.84:0/2679480616 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ba0144620 con 0x7f0ba01457f0 2024-03-29T10:45:26.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.042+0000 7f0b9cff9640 1 -- 172.21.15.84:0/2679480616 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0b880018e0 con 0x7f0ba01457f0 2024-03-29T10:45:26.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.042+0000 7f0b9cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:26.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.042+0000 7f0b9cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:26.044 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.042+0000 7f0b9cff9640 10 monclient(hunting): dump: 2024-03-29T10:45:26.044 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:26.044 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:26.044 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:26.044 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:26.044 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:26.044 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:26.044 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:26.044 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:26.045 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:26.045 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:26.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.042+0000 7f0b9cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:26.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.042+0000 7f0b9cff9640 1 -- 172.21.15.84:0/2679480616 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2619361243 0 0) 0x7f0ba0144340 con 0x7f0ba00a9d70 2024-03-29T10:45:26.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.042+0000 7f0b9cff9640 1 -- 172.21.15.84:0/2679480616 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b80004eb0 con 0x7f0ba00a9d70 2024-03-29T10:45:26.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.042+0000 7f0b9cff9640 1 -- 172.21.15.84:0/2679480616 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0b98001b90 con 0x7f0ba00a9d70 2024-03-29T10:45:26.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.042+0000 7f0b9cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:26.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0b9cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:26.045 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0b9cff9640 10 monclient(hunting): dump: 2024-03-29T10:45:26.045 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:26.045 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:26.045 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:26.045 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:26.045 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:26.046 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:26.046 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:26.046 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:26.046 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:26.046 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:26.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0b9cff9640 1 -- 172.21.15.84:0/2679480616 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 204760879 0 0) 0x7f0ba0145230 con 0x7f0ba013f650 2024-03-29T10:45:26.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0b9cff9640 1 -- 172.21.15.84:0/2679480616 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ba0144340 con 0x7f0ba013f650 2024-03-29T10:45:26.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0b9cff9640 1 -- 172.21.15.84:0/2679480616 <== mon.1 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3880716264 0 0) 0x7f0ba0144620 con 0x7f0ba01457f0 2024-03-29T10:45:26.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0b9cff9640 1 -- 172.21.15.84:0/2679480616 >> v1:172.21.15.84:6790/0 conn(0x7f0ba00a9d70 legacy=0x7f0ba0144ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:26.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0b9cff9640 1 -- 172.21.15.84:0/2679480616 >> v1:172.21.15.84:6789/0 conn(0x7f0ba013f650 legacy=0x7f0ba01e74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:26.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0b9cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:45:26.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0b9cff9640 1 monclient: found mon.b 2024-03-29T10:45:26.046 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0b9cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:26.047 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0b9cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:45:26.047 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0b9cff9640 1 -- 172.21.15.84:0/2679480616 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba013fda0 con 0x7f0ba01457f0 2024-03-29T10:45:26.047 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0b9cff9640 10 monclient: _finish_auth 0 2024-03-29T10:45:26.047 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0b9cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:45:26.047 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0b9cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:26.047 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0ba663c640 5 monclient: authenticate success, global_id 4528 2024-03-29T10:45:26.047 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0ba663c640 10 monclient: _renew_subs 2024-03-29T10:45:26.047 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0ba663c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:45:26.047 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0ba663c640 1 -- 172.21.15.84:0/2679480616 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ba0140150 con 0x7f0ba01457f0 2024-03-29T10:45:26.047 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0ba663c640 10 client.4528.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:26.047 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0ba663c640 10 monclient: _renew_subs 2024-03-29T10:45:26.047 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0ba663c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:45:26.047 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0ba663c640 1 -- 172.21.15.84:0/2679480616 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba01d9e10 con 0x7f0ba01457f0 2024-03-29T10:45:26.047 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0ba663c640 1 librados: init done 2024-03-29T10:45:26.047 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0b9cff9640 1 -- 172.21.15.84:0/2679480616 <== mon.1 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b880035f0 con 0x7f0ba01457f0 2024-03-29T10:45:26.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0b9cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:26.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0b9e7fc640 4 set_mon_vals no callback set 2024-03-29T10:45:26.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0b9cff9640 1 -- 172.21.15.84:0/2679480616 <== mon.1 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f0b88003940 con 0x7f0ba01457f0 2024-03-29T10:45:26.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0b9cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:26.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0b9cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:45:26.048 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.043+0000 7f0b9cff9640 10 monclient: dump: 2024-03-29T10:45:26.048 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:26.048 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:26.048 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:26.048 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:26.048 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:26.048 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:26.048 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:26.048 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:26.049 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:26.049 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:26.049 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.045+0000 7f0ba663c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:26.049 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.045+0000 7f0ba663c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:26.049 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.045+0000 7f0ba663c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:45:26.049 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.045+0000 7f0ba663c640 1 -- 172.21.15.84:0/2679480616 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b84005080 con 0x7f0ba01457f0 2024-03-29T10:45:26.049 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.045+0000 7f0b9cff9640 1 -- 172.21.15.84:0/2679480616 <== mon.1 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f0b88003aa0 con 0x7f0ba01457f0 2024-03-29T10:45:26.049 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.046+0000 7f0b9cff9640 1 -- 172.21.15.84:0/2679480616 <== mon.1 v1:172.21.15.86:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1523682299 0 0) 0x7f0b88084960 con 0x7f0ba01457f0 2024-03-29T10:45:26.049 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.046+0000 7f0b9cff9640 10 client.4528.objecter ms_dispatch 0x7f0ba0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:45:26.049 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.046+0000 7f0b9cff9640 3 client.4528.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:45:26.049 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.046+0000 7f0b9cff9640 3 client.4528.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:45:26.049 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.046+0000 7f0b9cff9640 20 client.4528.objecter dump_active .. 0 homeless 2024-03-29T10:45:26.049 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.046+0000 7f0b9ffff640 10 monclient: get_auth_request con 0x7f0b8006e010 auth_method 0 2024-03-29T10:45:26.049 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.046+0000 7f0b9cff9640 10 client.4528.objecter ms_handle_connect 0x7f0b8006e010 2024-03-29T10:45:26.050 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.054+0000 7f0b9cff9640 1 -- 172.21.15.84:0/2679480616 <== mon.1 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b88050240 con 0x7f0ba01457f0 2024-03-29T10:45:26.050 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.054+0000 7f0b9cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:26.050 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.054+0000 7f0b9cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:26.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.373+0000 7f0ba663c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:26.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.373+0000 7f0ba663c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:26.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.373+0000 7f0ba663c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.86:6789/0 2024-03-29T10:45:26.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.373+0000 7f0ba663c640 1 -- 172.21.15.84:0/2679480616 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b84005330 con 0x7f0ba01457f0 2024-03-29T10:45:26.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.374+0000 7f0b9cff9640 1 -- 172.21.15.84:0/2679480616 <== mon.1 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 879675878) 0x7f0b88052170 con 0x7f0ba01457f0 2024-03-29T10:45:26.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.374+0000 7f0b9cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:26.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.374+0000 7f0b9cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:26.357 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:45:26.357 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":11,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:44:56.255219+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:56.255219+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:26.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.377+0000 7f0ba663c640 10 librados: watch_flush enter 2024-03-29T10:45:26.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.377+0000 7f0ba663c640 10 librados: watch_flush exit 2024-03-29T10:45:26.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.377+0000 7f0ba663c640 20 client.4528.objecter shutdown clearing up homeless session... 2024-03-29T10:45:26.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.377+0000 7f0ba663c640 10 client.4528.objecter successfully canceled tick 2024-03-29T10:45:26.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.377+0000 7f0ba663c640 1 -- 172.21.15.84:0/2679480616 >> v1:172.21.15.86:6816/3337225337 conn(0x7f0b8006e010 legacy=0x7f0b800704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:26.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.377+0000 7f0ba663c640 10 monclient: shutdown 2024-03-29T10:45:26.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.377+0000 7f0ba663c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:26.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.377+0000 7f0ba663c640 1 -- 172.21.15.84:0/2679480616 >> v1:172.21.15.86:6789/0 conn(0x7f0ba01457f0 legacy=0x7f0ba01eeaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:26.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.377+0000 7f0ba663c640 1 -- 172.21.15.84:0/2679480616 shutdown_connections 2024-03-29T10:45:26.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.377+0000 7f0ba663c640 1 -- 172.21.15.84:0/2679480616 >> 172.21.15.84:0/2679480616 conn(0x7f0ba0066a00 msgr2=0x7f0ba01daa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:26.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.377+0000 7f0ba663c640 1 -- 172.21.15.84:0/2679480616 shutdown_connections 2024-03-29T10:45:26.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.377+0000 7f0ba663c640 1 -- 172.21.15.84:0/2679480616 wait complete. 2024-03-29T10:45:26.361 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:26.378+0000 7f0ba663c640 1 librados: shutdown 2024-03-29T10:45:26.375 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:33.182 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:33.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.344+0000 7fbd29cd3640 10 monclient: get_monmap_and_config 2024-03-29T10:45:33.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.344+0000 7fbd29cd3640 10 monclient: build_initial_monmap 2024-03-29T10:45:33.327 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.344+0000 7fbd29cd3640 10 monclient: monmap: 2024-03-29T10:45:33.327 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:45:33.327 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:33.327 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:45:33.345260+0000 2024-03-29T10:45:33.327 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:45:33.345260+0000 2024-03-29T10:45:33.327 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:33.327 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:33.328 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:45:33.328 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:45:33.328 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:45:33.328 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:33.331 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.348+0000 7fbd29cd3640 1 Processor -- start 2024-03-29T10:45:33.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.349+0000 7fbd29cd3640 1 -- start start 2024-03-29T10:45:33.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.349+0000 7fbd29cd3640 10 monclient: init 2024-03-29T10:45:33.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.350+0000 7fbd29cd3640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:33.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.350+0000 7fbd29cd3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:45:33.332 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.350+0000 7fbd29cd3640 10 monclient(hunting): picked mon.noname-b con 0x7fbd240b0770 addr v1:172.21.15.84:6790/0 2024-03-29T10:45:33.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.350+0000 7fbd29cd3640 10 monclient(hunting): picked mon.noname-a con 0x7fbd2414a700 addr v1:172.21.15.84:6789/0 2024-03-29T10:45:33.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.350+0000 7fbd29cd3640 10 monclient(hunting): picked mon.noname-c con 0x7fbd240ac5a0 addr v1:172.21.15.86:6789/0 2024-03-29T10:45:33.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.350+0000 7fbd29cd3640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd240aba70 con 0x7fbd2414a700 2024-03-29T10:45:33.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.350+0000 7fbd29cd3640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd240aecb0 con 0x7fbd240ac5a0 2024-03-29T10:45:33.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.350+0000 7fbd29cd3640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd240af150 con 0x7fbd240b0770 2024-03-29T10:45:33.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.350+0000 7fbd29cd3640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:33.333 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.350+0000 7fbd29cd3640 10 monclient(hunting): authenticate will time out at 1435.613267s 2024-03-29T10:45:33.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.350+0000 7fbd237fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7fbd240b0770 0x7fbd24149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:44206/0 (socket says 172.21.15.84:44206) 2024-03-29T10:45:33.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.351+0000 7fbd237fe640 1 -- 172.21.15.84:0/2779811825 learned_addr learned my addr 172.21.15.84:0/2779811825 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:45:33.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.351+0000 7fbd227fc640 1 -- 172.21.15.84:0/2779811825 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2125025009 0 0) 0x7fbd240af150 con 0x7fbd240b0770 2024-03-29T10:45:33.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.351+0000 7fbd227fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:33.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.351+0000 7fbd227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:33.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.351+0000 7fbd227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:33.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.351+0000 7fbd227fc640 10 monclient(hunting): my global_id is 4694 2024-03-29T10:45:33.335 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.351+0000 7fbd227fc640 1 -- 172.21.15.84:0/2779811825 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd0c003520 con 0x7fbd240b0770 2024-03-29T10:45:33.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.352+0000 7fbd227fc640 1 -- 172.21.15.84:0/2779811825 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 836062342 0 0) 0x7fbd0c003520 con 0x7fbd240b0770 2024-03-29T10:45:33.336 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.352+0000 7fbd227fc640 1 -- 172.21.15.84:0/2779811825 >> v1:172.21.15.86:6789/0 conn(0x7fbd240ac5a0 legacy=0x7fbd24151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:33.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.353+0000 7fbd227fc640 1 -- 172.21.15.84:0/2779811825 >> v1:172.21.15.84:6789/0 conn(0x7fbd2414a700 legacy=0x7fbd240abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:33.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.353+0000 7fbd227fc640 10 monclient: _finish_hunting 0 2024-03-29T10:45:33.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.353+0000 7fbd227fc640 1 monclient: found mon.noname-b 2024-03-29T10:45:33.338 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.353+0000 7fbd227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:33.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.354+0000 7fbd227fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:45:33.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.354+0000 7fbd227fc640 1 -- 172.21.15.84:0/2779811825 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd240ac9d0 con 0x7fbd240b0770 2024-03-29T10:45:33.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.354+0000 7fbd227fc640 10 monclient: _finish_auth 0 2024-03-29T10:45:33.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.354+0000 7fbd227fc640 10 monclient: _check_auth_tickets 2024-03-29T10:45:33.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.354+0000 7fbd227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:33.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.354+0000 7fbd227fc640 1 -- 172.21.15.84:0/2779811825 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fbd1c001a10 con 0x7fbd240b0770 2024-03-29T10:45:33.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.354+0000 7fbd227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:33.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.354+0000 7fbd227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:33.339 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.354+0000 7fbd227fc640 10 monclient: dump: 2024-03-29T10:45:33.339 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:33.339 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:33.339 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:33.340 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:33.340 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:33.340 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:33.340 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:33.340 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:33.340 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:33.340 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:33.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.354+0000 7fbd29cd3640 5 monclient: authenticate success, global_id 4694 2024-03-29T10:45:33.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.354+0000 7fbd29cd3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:33.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.354+0000 7fbd227fc640 1 -- 172.21.15.84:0/2779811825 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd1c001be0 con 0x7fbd240b0770 2024-03-29T10:45:33.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.354+0000 7fbd227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:33.340 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.354+0000 7fbd227fc640 1 -- 172.21.15.84:0/2779811825 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fbd1c002030 con 0x7fbd240b0770 2024-03-29T10:45:33.341 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.354+0000 7fbd227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:33.341 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.354+0000 7fbd227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:33.341 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.354+0000 7fbd227fc640 10 monclient: dump: 2024-03-29T10:45:33.341 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:33.341 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:33.341 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:33.341 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:33.341 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:33.341 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:33.341 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:33.341 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:33.341 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:33.341 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:33.341 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.355+0000 7fbd29cd3640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:33.342 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.355+0000 7fbd29cd3640 4 set_mon_vals no callback set 2024-03-29T10:45:33.342 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.356+0000 7fbd29cd3640 10 monclient: shutdown 2024-03-29T10:45:33.342 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.356+0000 7fbd29cd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:33.342 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.356+0000 7fbd29cd3640 1 -- 172.21.15.84:0/2779811825 >> v1:172.21.15.84:6790/0 conn(0x7fbd240b0770 legacy=0x7fbd24149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:33.342 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.358+0000 7fbd29cd3640 1 -- 172.21.15.84:0/2779811825 shutdown_connections 2024-03-29T10:45:33.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.358+0000 7fbd29cd3640 1 -- 172.21.15.84:0/2779811825 >> 172.21.15.84:0/2779811825 conn(0x7fbd2406b660 msgr2=0x7fbd2406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:33.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.359+0000 7fbd29cd3640 1 -- 172.21.15.84:0/2779811825 shutdown_connections 2024-03-29T10:45:33.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.360+0000 7fbd29cd3640 1 -- 172.21.15.84:0/2779811825 wait complete. 2024-03-29T10:45:33.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.360+0000 7fbd29cd3640 10 monclient: build_initial_monmap 2024-03-29T10:45:33.343 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.361+0000 7fbd29cd3640 10 monclient: monmap: 2024-03-29T10:45:33.343 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:45:33.343 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:33.344 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:45:33.344 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:45:33.344 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:33.344 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:33.344 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:45:33.344 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:45:33.344 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:45:33.344 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:33.345 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.361+0000 7fbd29cd3640 1 librados: starting msgr at 2024-03-29T10:45:33.345 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.361+0000 7fbd29cd3640 1 librados: starting objecter 2024-03-29T10:45:33.345 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.361+0000 7fbd29cd3640 1 Processor -- start 2024-03-29T10:45:33.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.362+0000 7fbd29cd3640 1 -- start start 2024-03-29T10:45:33.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.362+0000 7fbd29cd3640 1 librados: setting wanted keys 2024-03-29T10:45:33.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.362+0000 7fbd29cd3640 1 librados: calling monclient init 2024-03-29T10:45:33.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.362+0000 7fbd29cd3640 10 monclient: init 2024-03-29T10:45:33.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.363+0000 7fbd29cd3640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:33.346 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.363+0000 7fbd29cd3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:45:33.347 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.363+0000 7fbd29cd3640 10 monclient(hunting): picked mon.noname-b con 0x7fbd240ac5a0 addr v1:172.21.15.86:6789/0 2024-03-29T10:45:33.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.365+0000 7fbd29cd3640 10 monclient(hunting): picked mon.noname-a con 0x7fbd240b0770 addr v1:172.21.15.84:6789/0 2024-03-29T10:45:33.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.369+0000 7fbd29cd3640 10 monclient(hunting): picked mon.noname-c con 0x7fbd2414a700 addr v1:172.21.15.84:6790/0 2024-03-29T10:45:33.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.369+0000 7fbd29cd3640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd2412a310 con 0x7fbd240b0770 2024-03-29T10:45:33.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.369+0000 7fbd29cd3640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd24127e50 con 0x7fbd240ac5a0 2024-03-29T10:45:33.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.369+0000 7fbd29cd3640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd24147ac0 con 0x7fbd2414a700 2024-03-29T10:45:33.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.369+0000 7fbd29cd3640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:33.354 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.369+0000 7fbd29cd3640 10 monclient(hunting): authenticate will time out at 1435.632201s 2024-03-29T10:45:33.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.369+0000 7fbd237fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fbd240ac5a0 0x7fbd24146ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.84:52555/0 (socket says 172.21.15.84:52555) 2024-03-29T10:45:33.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.369+0000 7fbd237fe640 1 -- 172.21.15.84:0/271233542 learned_addr learned my addr 172.21.15.84:0/271233542 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:45:33.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.369+0000 7fbd0bfff640 10 client.?.objecter ms_handle_connect 0x7fbd240ac5a0 2024-03-29T10:45:33.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.369+0000 7fbd0bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:33.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.369+0000 7fbd0bfff640 10 client.?.objecter ms_handle_connect 0x7fbd2414a700 2024-03-29T10:45:33.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.369+0000 7fbd0bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:33.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.369+0000 7fbd0bfff640 10 client.?.objecter ms_handle_connect 0x7fbd240b0770 2024-03-29T10:45:33.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.369+0000 7fbd0bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:33.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.369+0000 7fbd0bfff640 1 -- 172.21.15.84:0/271233542 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1402319860 0 0) 0x7fbd24147ac0 con 0x7fbd2414a700 2024-03-29T10:45:33.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.369+0000 7fbd0bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:33.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.369+0000 7fbd0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:33.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.369+0000 7fbd0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:33.355 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.369+0000 7fbd0bfff640 10 monclient(hunting): my global_id is 4697 2024-03-29T10:45:33.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.369+0000 7fbd0bfff640 1 -- 172.21.15.84:0/271233542 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd10004eb0 con 0x7fbd2414a700 2024-03-29T10:45:33.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.369+0000 7fbd0bfff640 1 -- 172.21.15.84:0/271233542 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4127828819 0 0) 0x7fbd24127e50 con 0x7fbd240ac5a0 2024-03-29T10:45:33.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.369+0000 7fbd0bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:33.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.369+0000 7fbd0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:33.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.369+0000 7fbd0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:33.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.369+0000 7fbd0bfff640 10 monclient(hunting): my global_id is 4531 2024-03-29T10:45:33.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.369+0000 7fbd0bfff640 1 -- 172.21.15.84:0/271233542 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd24147ac0 con 0x7fbd240ac5a0 2024-03-29T10:45:33.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.370+0000 7fbd0bfff640 1 -- 172.21.15.84:0/271233542 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4219045941 0 0) 0x7fbd2412a310 con 0x7fbd240b0770 2024-03-29T10:45:33.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.370+0000 7fbd0bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:33.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.370+0000 7fbd0bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:33.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.370+0000 7fbd0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:33.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.370+0000 7fbd0bfff640 10 monclient(hunting): my global_id is 4596 2024-03-29T10:45:33.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.370+0000 7fbd0bfff640 1 -- 172.21.15.84:0/271233542 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd24127e50 con 0x7fbd240b0770 2024-03-29T10:45:33.356 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.370+0000 7fbd0bfff640 1 -- 172.21.15.84:0/271233542 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2462714326 0 0) 0x7fbd10004eb0 con 0x7fbd2414a700 2024-03-29T10:45:33.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.370+0000 7fbd0bfff640 1 -- 172.21.15.84:0/271233542 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd2412a310 con 0x7fbd2414a700 2024-03-29T10:45:33.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.370+0000 7fbd0bfff640 1 -- 172.21.15.84:0/271233542 <== mon.1 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2860203948 0 0) 0x7fbd24147ac0 con 0x7fbd240ac5a0 2024-03-29T10:45:33.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.370+0000 7fbd0bfff640 1 -- 172.21.15.84:0/271233542 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd10004eb0 con 0x7fbd240ac5a0 2024-03-29T10:45:33.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.370+0000 7fbd0bfff640 1 -- 172.21.15.84:0/271233542 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fbd18001720 con 0x7fbd2414a700 2024-03-29T10:45:33.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.370+0000 7fbd0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:33.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.370+0000 7fbd0bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:33.357 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.370+0000 7fbd0bfff640 10 monclient(hunting): dump: 2024-03-29T10:45:33.357 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:33.357 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:33.357 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:33.357 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:33.357 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:33.357 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:33.358 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:33.358 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:33.358 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:33.358 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:33.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.370+0000 7fbd0bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:33.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.370+0000 7fbd0bfff640 1 -- 172.21.15.84:0/271233542 <== mon.1 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fbd1c001ef0 con 0x7fbd240ac5a0 2024-03-29T10:45:33.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.370+0000 7fbd0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:33.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.370+0000 7fbd0bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:45:33.358 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.370+0000 7fbd0bfff640 10 monclient(hunting): dump: 2024-03-29T10:45:33.358 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:33.358 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:33.358 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:33.358 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:33.359 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:33.359 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:33.359 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:33.359 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:33.359 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:33.359 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:33.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.370+0000 7fbd0bfff640 1 -- 172.21.15.84:0/271233542 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3190430887 0 0) 0x7fbd2412a310 con 0x7fbd2414a700 2024-03-29T10:45:33.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.371+0000 7fbd0bfff640 1 -- 172.21.15.84:0/271233542 >> v1:172.21.15.86:6789/0 conn(0x7fbd240ac5a0 legacy=0x7fbd24146ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:33.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.371+0000 7fbd0bfff640 1 -- 172.21.15.84:0/271233542 >> v1:172.21.15.84:6789/0 conn(0x7fbd240b0770 legacy=0x7fbd24128370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:33.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.371+0000 7fbd0bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:45:33.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.371+0000 7fbd0bfff640 1 monclient: found mon.c 2024-03-29T10:45:33.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.371+0000 7fbd0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:33.359 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.371+0000 7fbd0bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:45:33.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.371+0000 7fbd0bfff640 1 -- 172.21.15.84:0/271233542 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd24147c50 con 0x7fbd2414a700 2024-03-29T10:45:33.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.371+0000 7fbd0bfff640 10 monclient: _finish_auth 0 2024-03-29T10:45:33.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.371+0000 7fbd0bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:45:33.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.371+0000 7fbd0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:33.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.371+0000 7fbd29cd3640 5 monclient: authenticate success, global_id 4697 2024-03-29T10:45:33.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.371+0000 7fbd29cd3640 10 monclient: _renew_subs 2024-03-29T10:45:33.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.371+0000 7fbd29cd3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:45:33.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.371+0000 7fbd29cd3640 1 -- 172.21.15.84:0/271233542 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd241df590 con 0x7fbd2414a700 2024-03-29T10:45:33.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.371+0000 7fbd29cd3640 10 client.4697.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:33.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.371+0000 7fbd29cd3640 10 monclient: _renew_subs 2024-03-29T10:45:33.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.371+0000 7fbd29cd3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:45:33.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.371+0000 7fbd29cd3640 1 -- 172.21.15.84:0/271233542 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd241477c0 con 0x7fbd2414a700 2024-03-29T10:45:33.360 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.371+0000 7fbd29cd3640 1 librados: init done 2024-03-29T10:45:33.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.372+0000 7fbd09ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:33.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.372+0000 7fbd09ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:33.362 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.372+0000 7fbd09ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:45:33.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.372+0000 7fbd09ffb640 1 -- 172.21.15.84:0/271233542 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd04005080 con 0x7fbd2414a700 2024-03-29T10:45:33.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.374+0000 7fbd0bfff640 1 -- 172.21.15.84:0/271233542 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd18001880 con 0x7fbd2414a700 2024-03-29T10:45:33.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.375+0000 7fbd0bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:33.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.375+0000 7fbd0bfff640 1 -- 172.21.15.84:0/271233542 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7fbd18003690 con 0x7fbd2414a700 2024-03-29T10:45:33.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.375+0000 7fbd0bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:33.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.375+0000 7fbd0bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:33.363 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.375+0000 7fbd0bfff640 10 monclient: dump: 2024-03-29T10:45:33.363 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:33.363 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:33.363 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:33.363 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:33.363 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:33.363 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:33.363 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:33.364 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:33.364 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:33.364 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:33.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.375+0000 7fbd21ffb640 4 set_mon_vals no callback set 2024-03-29T10:45:33.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.376+0000 7fbd0bfff640 1 -- 172.21.15.84:0/271233542 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7fbd18019660 con 0x7fbd2414a700 2024-03-29T10:45:33.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.377+0000 7fbd0bfff640 1 -- 172.21.15.84:0/271233542 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1523682299 0 0) 0x7fbd180836b0 con 0x7fbd2414a700 2024-03-29T10:45:33.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.377+0000 7fbd0bfff640 10 client.4697.objecter ms_dispatch 0x7fbd24000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:45:33.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.377+0000 7fbd0bfff640 3 client.4697.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:45:33.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.377+0000 7fbd0bfff640 3 client.4697.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:45:33.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.377+0000 7fbd0bfff640 20 client.4697.objecter dump_active .. 0 homeless 2024-03-29T10:45:33.364 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.377+0000 7fbd237fe640 10 monclient: get_auth_request con 0x7fbd1006dd60 auth_method 0 2024-03-29T10:45:33.365 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.381+0000 7fbd0bfff640 10 client.4697.objecter ms_handle_connect 0x7fbd1006dd60 2024-03-29T10:45:33.365 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.383+0000 7fbd0bfff640 1 -- 172.21.15.84:0/271233542 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbd1804edc0 con 0x7fbd2414a700 2024-03-29T10:45:33.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.383+0000 7fbd0bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:33.366 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.383+0000 7fbd0bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:33.730 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.747+0000 7fbd09ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:33.730 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.747+0000 7fbd09ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:33.730 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.747+0000 7fbd09ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:45:33.730 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.747+0000 7fbd09ffb640 1 -- 172.21.15.84:0/271233542 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd04005330 con 0x7fbd2414a700 2024-03-29T10:45:33.730 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.748+0000 7fbd0bfff640 1 -- 172.21.15.84:0/271233542 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 879675878) 0x7fbd18050ec0 con 0x7fbd2414a700 2024-03-29T10:45:33.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.748+0000 7fbd0bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:33.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.748+0000 7fbd0bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:33.731 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:45:33.731 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":11,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:44:56.255219+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:56.255219+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:33.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.752+0000 7fbd09ffb640 10 librados: watch_flush enter 2024-03-29T10:45:33.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.752+0000 7fbd09ffb640 10 librados: watch_flush exit 2024-03-29T10:45:33.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.752+0000 7fbd09ffb640 20 client.4697.objecter shutdown clearing up homeless session... 2024-03-29T10:45:33.734 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.752+0000 7fbd09ffb640 10 client.4697.objecter successfully canceled tick 2024-03-29T10:45:33.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.752+0000 7fbd09ffb640 1 -- 172.21.15.84:0/271233542 >> v1:172.21.15.86:6816/3337225337 conn(0x7fbd1006dd60 legacy=0x7fbd10070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:33.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.752+0000 7fbd09ffb640 10 monclient: shutdown 2024-03-29T10:45:33.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.752+0000 7fbd09ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:33.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.752+0000 7fbd09ffb640 1 -- 172.21.15.84:0/271233542 >> v1:172.21.15.84:6790/0 conn(0x7fbd2414a700 legacy=0x7fbd24128a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:33.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.752+0000 7fbd09ffb640 1 -- 172.21.15.84:0/271233542 shutdown_connections 2024-03-29T10:45:33.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.752+0000 7fbd09ffb640 1 -- 172.21.15.84:0/271233542 >> 172.21.15.84:0/271233542 conn(0x7fbd2406b660 msgr2=0x7fbd240b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:33.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.753+0000 7fbd09ffb640 1 -- 172.21.15.84:0/271233542 shutdown_connections 2024-03-29T10:45:33.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.753+0000 7fbd09ffb640 1 -- 172.21.15.84:0/271233542 wait complete. 2024-03-29T10:45:33.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:33.753+0000 7fbd09ffb640 1 librados: shutdown 2024-03-29T10:45:33.750 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:40.560 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:40.714 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.729+0000 7ff13db46640 10 monclient: get_monmap_and_config 2024-03-29T10:45:40.714 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.729+0000 7ff13db46640 10 monclient: build_initial_monmap 2024-03-29T10:45:40.714 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.729+0000 7ff13db46640 10 monclient: monmap: 2024-03-29T10:45:40.714 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:45:40.714 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:40.715 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:45:40.731075+0000 2024-03-29T10:45:40.715 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:45:40.731075+0000 2024-03-29T10:45:40.715 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:40.715 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:40.715 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:45:40.715 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:45:40.715 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:45:40.715 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:40.716 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.731+0000 7ff13db46640 1 Processor -- start 2024-03-29T10:45:40.716 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.731+0000 7ff13db46640 1 -- start start 2024-03-29T10:45:40.716 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.731+0000 7ff13db46640 10 monclient: init 2024-03-29T10:45:40.716 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.732+0000 7ff13db46640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:40.716 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.732+0000 7ff13db46640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:45:40.716 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.732+0000 7ff13db46640 10 monclient(hunting): picked mon.noname-b con 0x7ff1380a9750 addr v1:172.21.15.84:6790/0 2024-03-29T10:45:40.719 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.732+0000 7ff13db46640 10 monclient(hunting): picked mon.noname-a con 0x7ff1380ae220 addr v1:172.21.15.84:6789/0 2024-03-29T10:45:40.719 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.732+0000 7ff13db46640 10 monclient(hunting): picked mon.noname-c con 0x7ff13813b9d0 addr v1:172.21.15.86:6789/0 2024-03-29T10:45:40.719 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.732+0000 7ff13db46640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff138142490 con 0x7ff1380ae220 2024-03-29T10:45:40.719 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.732+0000 7ff13db46640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1380ac130 con 0x7ff13813b9d0 2024-03-29T10:45:40.719 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.732+0000 7ff13db46640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1380ac410 con 0x7ff1380a9750 2024-03-29T10:45:40.719 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.732+0000 7ff13db46640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:40.719 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.732+0000 7ff13db46640 10 monclient(hunting): authenticate will time out at 1442.996826s 2024-03-29T10:45:40.719 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.734+0000 7ff13cb44640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7ff1380a9750 0x7ff138141d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:40668/0 (socket says 172.21.15.84:40668) 2024-03-29T10:45:40.719 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.734+0000 7ff13cb44640 1 -- 172.21.15.84:0/1675568076 learned_addr learned my addr 172.21.15.84:0/1675568076 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:45:40.719 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.735+0000 7ff1377fe640 1 -- 172.21.15.84:0/1675568076 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1627085782 0 0) 0x7ff1380ac410 con 0x7ff1380a9750 2024-03-29T10:45:40.719 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.735+0000 7ff1377fe640 10 monclient(hunting): _negotiate 2024-03-29T10:45:40.719 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.735+0000 7ff1377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:40.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.735+0000 7ff1377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:40.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.735+0000 7ff1377fe640 10 monclient(hunting): my global_id is 4700 2024-03-29T10:45:40.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.735+0000 7ff1377fe640 1 -- 172.21.15.84:0/1675568076 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff128003520 con 0x7ff1380a9750 2024-03-29T10:45:40.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.735+0000 7ff1377fe640 1 -- 172.21.15.84:0/1675568076 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1947037721 0 0) 0x7ff128003520 con 0x7ff1380a9750 2024-03-29T10:45:40.720 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.735+0000 7ff1377fe640 1 -- 172.21.15.84:0/1675568076 >> v1:172.21.15.86:6789/0 conn(0x7ff13813b9d0 legacy=0x7ff138149930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:40.724 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.738+0000 7ff1377fe640 1 -- 172.21.15.84:0/1675568076 >> v1:172.21.15.84:6789/0 conn(0x7ff1380ae220 legacy=0x7ff13813b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:45:40.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.738+0000 7ff1377fe640 10 monclient: _finish_hunting 0 2024-03-29T10:45:40.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.738+0000 7ff1377fe640 1 monclient: found mon.noname-b 2024-03-29T10:45:40.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.738+0000 7ff1377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:40.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.738+0000 7ff1377fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.84:6790/0 2024-03-29T10:45:40.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.738+0000 7ff1377fe640 1 -- 172.21.15.84:0/1675568076 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1380afa40 con 0x7ff1380a9750 2024-03-29T10:45:40.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.738+0000 7ff1377fe640 10 monclient: _finish_auth 0 2024-03-29T10:45:40.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.738+0000 7ff1377fe640 10 monclient: _check_auth_tickets 2024-03-29T10:45:40.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.738+0000 7ff1377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:40.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.738+0000 7ff1377fe640 1 -- 172.21.15.84:0/1675568076 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7ff124001690 con 0x7ff1380a9750 2024-03-29T10:45:40.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.738+0000 7ff1377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:40.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.738+0000 7ff1377fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:45:40.725 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.738+0000 7ff1377fe640 10 monclient: dump: 2024-03-29T10:45:40.725 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:40.725 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:40.726 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:40.726 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:40.726 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:40.726 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:40.726 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:40.726 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:40.726 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:40.726 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:40.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.739+0000 7ff13db46640 5 monclient: authenticate success, global_id 4700 2024-03-29T10:45:40.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.739+0000 7ff13db46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:40.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.739+0000 7ff1377fe640 1 -- 172.21.15.84:0/1675568076 <== mon.2 v1:172.21.15.84:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1240017f0 con 0x7ff1380a9750 2024-03-29T10:45:40.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.739+0000 7ff1377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:40.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.739+0000 7ff1377fe640 1 -- 172.21.15.84:0/1675568076 <== mon.2 v1:172.21.15.84:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7ff124001c40 con 0x7ff1380a9750 2024-03-29T10:45:40.726 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.739+0000 7ff1377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:40.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.739+0000 7ff1377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:40.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.739+0000 7ff1377fe640 10 monclient: dump: 2024-03-29T10:45:40.727 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:40.727 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:40.727 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:40.727 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:40.727 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:40.727 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:40.727 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:40.727 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:40.727 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:40.727 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:40.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.740+0000 7ff13db46640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:40.727 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.740+0000 7ff13db46640 4 set_mon_vals no callback set 2024-03-29T10:45:40.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.740+0000 7ff13db46640 10 monclient: shutdown 2024-03-29T10:45:40.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.741+0000 7ff13db46640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:40.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.741+0000 7ff13db46640 1 -- 172.21.15.84:0/1675568076 >> v1:172.21.15.84:6790/0 conn(0x7ff1380a9750 legacy=0x7ff138141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:40.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.741+0000 7ff13db46640 1 -- 172.21.15.84:0/1675568076 shutdown_connections 2024-03-29T10:45:40.728 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.741+0000 7ff13db46640 1 -- 172.21.15.84:0/1675568076 >> 172.21.15.84:0/1675568076 conn(0x7ff1380673e0 msgr2=0x7ff1380b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:40.730 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.741+0000 7ff13db46640 1 -- 172.21.15.84:0/1675568076 shutdown_connections 2024-03-29T10:45:40.730 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.741+0000 7ff13db46640 1 -- 172.21.15.84:0/1675568076 wait complete. 2024-03-29T10:45:40.730 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.742+0000 7ff13db46640 10 monclient: build_initial_monmap 2024-03-29T10:45:40.730 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.743+0000 7ff13db46640 10 monclient: monmap: 2024-03-29T10:45:40.730 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:45:40.730 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:40.730 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:45:40.730 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:45:40.731 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:40.731 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:40.731 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:45:40.731 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:45:40.731 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:45:40.731 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:40.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.743+0000 7ff13db46640 1 librados: starting msgr at 2024-03-29T10:45:40.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.743+0000 7ff13db46640 1 librados: starting objecter 2024-03-29T10:45:40.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.744+0000 7ff13db46640 1 Processor -- start 2024-03-29T10:45:40.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.744+0000 7ff13db46640 1 -- start start 2024-03-29T10:45:40.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.744+0000 7ff13db46640 1 librados: setting wanted keys 2024-03-29T10:45:40.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.744+0000 7ff13db46640 1 librados: calling monclient init 2024-03-29T10:45:40.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.744+0000 7ff13db46640 10 monclient: init 2024-03-29T10:45:40.731 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.745+0000 7ff13db46640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:40.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.745+0000 7ff13db46640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:45:40.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.745+0000 7ff13db46640 10 monclient(hunting): picked mon.noname-b con 0x7ff1380a9750 addr v1:172.21.15.86:6789/0 2024-03-29T10:45:40.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.745+0000 7ff13db46640 10 monclient(hunting): picked mon.noname-a con 0x7ff1380ae220 addr v1:172.21.15.84:6789/0 2024-03-29T10:45:40.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.745+0000 7ff13db46640 10 monclient(hunting): picked mon.noname-c con 0x7ff13813b9d0 addr v1:172.21.15.84:6790/0 2024-03-29T10:45:40.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.745+0000 7ff13db46640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff138070290 con 0x7ff1380ae220 2024-03-29T10:45:40.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.745+0000 7ff13db46640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff13806dba0 con 0x7ff1380a9750 2024-03-29T10:45:40.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.745+0000 7ff13db46640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff13806f8b0 con 0x7ff13813b9d0 2024-03-29T10:45:40.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.745+0000 7ff13db46640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:40.732 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.745+0000 7ff13db46640 10 monclient(hunting): authenticate will time out at 1443.009426s 2024-03-29T10:45:40.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.747+0000 7ff13d345640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7ff13813b9d0 0x7ff138070700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:42814/0 (socket says 172.21.15.84:42814) 2024-03-29T10:45:40.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.747+0000 7ff13d345640 1 -- 172.21.15.84:0/2893127369 learned_addr learned my addr 172.21.15.84:0/2893127369 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:45:40.735 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.748+0000 7ff1357fa640 10 client.?.objecter ms_handle_connect 0x7ff13813b9d0 2024-03-29T10:45:40.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.748+0000 7ff1357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:40.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.748+0000 7ff1357fa640 1 -- 172.21.15.84:0/2893127369 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3014020690 0 0) 0x7ff13806f8b0 con 0x7ff13813b9d0 2024-03-29T10:45:40.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.748+0000 7ff1357fa640 10 monclient(hunting): _negotiate 2024-03-29T10:45:40.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.748+0000 7ff1357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:40.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.748+0000 7ff1357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:40.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.748+0000 7ff1357fa640 10 monclient(hunting): my global_id is 4703 2024-03-29T10:45:40.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.748+0000 7ff1357fa640 1 -- 172.21.15.84:0/2893127369 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff11c004eb0 con 0x7ff13813b9d0 2024-03-29T10:45:40.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.748+0000 7ff1357fa640 10 client.?.objecter ms_handle_connect 0x7ff1380a9750 2024-03-29T10:45:40.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.748+0000 7ff1357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:40.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.748+0000 7ff1357fa640 1 -- 172.21.15.84:0/2893127369 <== mon.1 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2724933583 0 0) 0x7ff13806dba0 con 0x7ff1380a9750 2024-03-29T10:45:40.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.748+0000 7ff1357fa640 10 monclient(hunting): _negotiate 2024-03-29T10:45:40.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.749+0000 7ff1357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:40.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.749+0000 7ff1357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:40.736 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.749+0000 7ff1357fa640 10 monclient(hunting): my global_id is 4534 2024-03-29T10:45:40.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.749+0000 7ff1357fa640 1 -- 172.21.15.84:0/2893127369 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff13806f8b0 con 0x7ff1380a9750 2024-03-29T10:45:40.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.749+0000 7ff1357fa640 10 client.?.objecter ms_handle_connect 0x7ff1380ae220 2024-03-29T10:45:40.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.749+0000 7ff1357fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:40.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.749+0000 7ff1357fa640 1 -- 172.21.15.84:0/2893127369 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167030918 0 0) 0x7ff138070290 con 0x7ff1380ae220 2024-03-29T10:45:40.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.749+0000 7ff1357fa640 10 monclient(hunting): _negotiate 2024-03-29T10:45:40.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.749+0000 7ff1357fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:40.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.749+0000 7ff1357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:40.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.749+0000 7ff1357fa640 10 monclient(hunting): my global_id is 4599 2024-03-29T10:45:40.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.749+0000 7ff1357fa640 1 -- 172.21.15.84:0/2893127369 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff13806dba0 con 0x7ff1380ae220 2024-03-29T10:45:40.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.750+0000 7ff1357fa640 1 -- 172.21.15.84:0/2893127369 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2682098585 0 0) 0x7ff13806dba0 con 0x7ff1380ae220 2024-03-29T10:45:40.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.750+0000 7ff1357fa640 1 -- 172.21.15.84:0/2893127369 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff138070290 con 0x7ff1380ae220 2024-03-29T10:45:40.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.750+0000 7ff1357fa640 1 -- 172.21.15.84:0/2893127369 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7ff12c001510 con 0x7ff1380ae220 2024-03-29T10:45:40.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.750+0000 7ff1357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:40.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.750+0000 7ff1357fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:40.737 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.750+0000 7ff1357fa640 10 monclient(hunting): dump: 2024-03-29T10:45:40.738 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:40.738 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:40.738 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:40.738 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:40.738 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:40.738 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:40.738 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:40.738 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:40.738 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:40.738 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:40.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.750+0000 7ff1357fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:40.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.750+0000 7ff1357fa640 1 -- 172.21.15.84:0/2893127369 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1255498644 0 0) 0x7ff11c004eb0 con 0x7ff13813b9d0 2024-03-29T10:45:40.738 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.750+0000 7ff1357fa640 1 -- 172.21.15.84:0/2893127369 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff13806dba0 con 0x7ff13813b9d0 2024-03-29T10:45:40.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.750+0000 7ff1357fa640 1 -- 172.21.15.84:0/2893127369 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7ff130001a50 con 0x7ff13813b9d0 2024-03-29T10:45:40.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.750+0000 7ff1357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:40.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.750+0000 7ff1357fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:40.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.750+0000 7ff1357fa640 10 monclient(hunting): dump: 2024-03-29T10:45:40.739 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:40.739 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:40.739 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:40.739 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:40.739 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:40.739 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:40.739 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:40.739 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:40.739 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:40.739 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:40.739 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.750+0000 7ff1357fa640 1 -- 172.21.15.84:0/2893127369 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2202656941 0 0) 0x7ff13806dba0 con 0x7ff13813b9d0 2024-03-29T10:45:40.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.750+0000 7ff1357fa640 1 -- 172.21.15.84:0/2893127369 >> v1:172.21.15.86:6789/0 conn(0x7ff1380a9750 legacy=0x7ff1381ded90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:40.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.750+0000 7ff1357fa640 1 -- 172.21.15.84:0/2893127369 >> v1:172.21.15.84:6789/0 conn(0x7ff1380ae220 legacy=0x7ff1380b2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:40.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.750+0000 7ff1357fa640 10 monclient: _finish_hunting 0 2024-03-29T10:45:40.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.750+0000 7ff1357fa640 1 monclient: found mon.c 2024-03-29T10:45:40.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.750+0000 7ff1357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:40.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.750+0000 7ff1357fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:45:40.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.750+0000 7ff1357fa640 1 -- 172.21.15.84:0/2893127369 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff13806fb90 con 0x7ff13813b9d0 2024-03-29T10:45:40.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.750+0000 7ff1357fa640 10 monclient: _finish_auth 0 2024-03-29T10:45:40.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.750+0000 7ff1357fa640 10 monclient: _check_auth_tickets 2024-03-29T10:45:40.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.750+0000 7ff1357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:40.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.751+0000 7ff1357fa640 1 -- 172.21.15.84:0/2893127369 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff130001f60 con 0x7ff13813b9d0 2024-03-29T10:45:40.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.751+0000 7ff1357fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:40.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.751+0000 7ff1357fa640 1 -- 172.21.15.84:0/2893127369 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7ff130004d80 con 0x7ff13813b9d0 2024-03-29T10:45:40.740 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.751+0000 7ff1357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:40.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.751+0000 7ff1357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:40.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.751+0000 7ff1357fa640 10 monclient: dump: 2024-03-29T10:45:40.741 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:40.741 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:40.741 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:40.741 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:40.741 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:40.741 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:40.741 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:40.741 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:40.741 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:40.741 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:40.741 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.751+0000 7ff1367fc640 4 set_mon_vals no callback set 2024-03-29T10:45:40.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.752+0000 7ff13db46640 5 monclient: authenticate success, global_id 4703 2024-03-29T10:45:40.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.752+0000 7ff13db46640 10 monclient: _renew_subs 2024-03-29T10:45:40.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.752+0000 7ff13db46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:45:40.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.752+0000 7ff13db46640 1 -- 172.21.15.84:0/2893127369 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff13806fe70 con 0x7ff13813b9d0 2024-03-29T10:45:40.743 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.752+0000 7ff13db46640 10 client.4703.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:40.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.752+0000 7ff13db46640 10 monclient: _renew_subs 2024-03-29T10:45:40.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.752+0000 7ff13db46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:45:40.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.752+0000 7ff13db46640 1 -- 172.21.15.84:0/2893127369 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff13806e2f0 con 0x7ff13813b9d0 2024-03-29T10:45:40.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.752+0000 7ff13db46640 1 librados: init done 2024-03-29T10:45:40.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.753+0000 7ff13db46640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:40.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.753+0000 7ff13db46640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:40.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.753+0000 7ff13db46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:45:40.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.753+0000 7ff13db46640 1 -- 172.21.15.84:0/2893127369 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff13804f730 con 0x7ff13813b9d0 2024-03-29T10:45:40.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.754+0000 7ff1357fa640 1 -- 172.21.15.84:0/2893127369 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7ff13001ad50 con 0x7ff13813b9d0 2024-03-29T10:45:40.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.757+0000 7ff1357fa640 1 -- 172.21.15.84:0/2893127369 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1523682299 0 0) 0x7ff130084c40 con 0x7ff13813b9d0 2024-03-29T10:45:40.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.757+0000 7ff1357fa640 10 client.4703.objecter ms_dispatch 0x7ff138000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:45:40.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.757+0000 7ff1357fa640 3 client.4703.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:45:40.744 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.757+0000 7ff1357fa640 3 client.4703.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:45:40.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.757+0000 7ff1357fa640 20 client.4703.objecter dump_active .. 0 homeless 2024-03-29T10:45:40.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.758+0000 7ff13cb44640 10 monclient: get_auth_request con 0x7ff11c06df00 auth_method 0 2024-03-29T10:45:40.745 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.760+0000 7ff1357fa640 10 client.4703.objecter ms_handle_connect 0x7ff11c06df00 2024-03-29T10:45:40.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.764+0000 7ff1357fa640 1 -- 172.21.15.84:0/2893127369 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff130050580 con 0x7ff13813b9d0 2024-03-29T10:45:40.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.765+0000 7ff1357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:40.748 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:40.765+0000 7ff1357fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:41.065 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:41.079+0000 7ff13db46640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:41.065 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:41.079+0000 7ff13db46640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:41.065 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:41.079+0000 7ff13db46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:45:41.065 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:41.079+0000 7ff13db46640 1 -- 172.21.15.84:0/2893127369 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1380abb70 con 0x7ff13813b9d0 2024-03-29T10:45:41.065 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:41.081+0000 7ff1357fa640 1 -- 172.21.15.84:0/2893127369 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 879675878) 0x7ff1300842f0 con 0x7ff13813b9d0 2024-03-29T10:45:41.066 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:41.081+0000 7ff1357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:41.066 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:41.081+0000 7ff1357fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:41.066 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:45:41.066 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":11,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:44:56.255219+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:56.255219+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:41.069 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:41.085+0000 7ff13db46640 10 librados: watch_flush enter 2024-03-29T10:45:41.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:41.086+0000 7ff13db46640 10 librados: watch_flush exit 2024-03-29T10:45:41.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:41.086+0000 7ff13db46640 20 client.4703.objecter shutdown clearing up homeless session... 2024-03-29T10:45:41.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:41.086+0000 7ff13db46640 10 client.4703.objecter successfully canceled tick 2024-03-29T10:45:41.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:41.086+0000 7ff13db46640 1 -- 172.21.15.84:0/2893127369 >> v1:172.21.15.86:6816/3337225337 conn(0x7ff11c06df00 legacy=0x7ff11c0703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:41.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:41.086+0000 7ff13db46640 10 monclient: shutdown 2024-03-29T10:45:41.071 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:41.086+0000 7ff13db46640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:41.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:41.086+0000 7ff13db46640 1 -- 172.21.15.84:0/2893127369 >> v1:172.21.15.84:6790/0 conn(0x7ff13813b9d0 legacy=0x7ff138070700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:41.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:41.088+0000 7ff13db46640 1 -- 172.21.15.84:0/2893127369 shutdown_connections 2024-03-29T10:45:41.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:41.088+0000 7ff13db46640 1 -- 172.21.15.84:0/2893127369 >> 172.21.15.84:0/2893127369 conn(0x7ff1380673e0 msgr2=0x7ff138069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:41.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:41.088+0000 7ff13db46640 1 -- 172.21.15.84:0/2893127369 shutdown_connections 2024-03-29T10:45:41.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:41.088+0000 7ff13db46640 1 -- 172.21.15.84:0/2893127369 wait complete. 2024-03-29T10:45:41.072 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:41.088+0000 7ff13db46640 1 librados: shutdown 2024-03-29T10:45:41.088 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:47.896 DEBUG:teuthology.orchestra.run.smithi084:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:45:48.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.050+0000 7f446dd3d640 10 monclient: get_monmap_and_config 2024-03-29T10:45:48.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.050+0000 7f446dd3d640 10 monclient: build_initial_monmap 2024-03-29T10:45:48.033 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.050+0000 7f446dd3d640 10 monclient: monmap: 2024-03-29T10:45:48.033 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:45:48.033 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:48.033 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:45:48.051041+0000 2024-03-29T10:45:48.034 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:45:48.051041+0000 2024-03-29T10:45:48.034 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:48.034 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:48.034 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:45:48.034 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-c 2024-03-29T10:45:48.034 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-b 2024-03-29T10:45:48.034 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:48.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.051+0000 7f446dd3d640 1 Processor -- start 2024-03-29T10:45:48.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.051+0000 7f446dd3d640 1 -- start start 2024-03-29T10:45:48.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.051+0000 7f446dd3d640 10 monclient: init 2024-03-29T10:45:48.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.052+0000 7f446dd3d640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:48.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.052+0000 7f446dd3d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:45:48.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.052+0000 7f446dd3d640 10 monclient(hunting): picked mon.noname-b con 0x7f44680a9ac0 addr v1:172.21.15.84:6790/0 2024-03-29T10:45:48.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.052+0000 7f446dd3d640 10 monclient(hunting): picked mon.noname-c con 0x7f446813ee90 addr v1:172.21.15.86:6789/0 2024-03-29T10:45:48.036 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.052+0000 7f446dd3d640 10 monclient(hunting): picked mon.noname-a con 0x7f44680af900 addr v1:172.21.15.84:6789/0 2024-03-29T10:45:48.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.052+0000 7f446dd3d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44680afd10 con 0x7f44680af900 2024-03-29T10:45:48.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.052+0000 7f446dd3d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44680af6f0 con 0x7f446813ee90 2024-03-29T10:45:48.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.052+0000 7f446dd3d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44680aac80 con 0x7f44680a9ac0 2024-03-29T10:45:48.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.052+0000 7f446dd3d640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:48.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.052+0000 7f446dd3d640 10 monclient(hunting): authenticate will time out at 1450.315217s 2024-03-29T10:45:48.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.052+0000 7f44677fe640 1 --1- >> v1:172.21.15.84:6790/0 conn(0x7f44680a9ac0 0x7f44680adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.84:6790/0 says I am v1:172.21.15.84:59536/0 (socket says 172.21.15.84:59536) 2024-03-29T10:45:48.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.052+0000 7f44677fe640 1 -- 172.21.15.84:0/4074538751 learned_addr learned my addr 172.21.15.84:0/4074538751 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:45:48.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.052+0000 7f44667fc640 1 -- 172.21.15.84:0/4074538751 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623810431 0 0) 0x7f44680aac80 con 0x7f44680a9ac0 2024-03-29T10:45:48.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.052+0000 7f44667fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:48.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.052+0000 7f44667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:48.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.052+0000 7f44667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:48.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.052+0000 7f44667fc640 10 monclient(hunting): my global_id is 4706 2024-03-29T10:45:48.037 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.052+0000 7f44667fc640 1 -- 172.21.15.84:0/4074538751 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4450003520 con 0x7f44680a9ac0 2024-03-29T10:45:48.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.052+0000 7f44667fc640 1 -- 172.21.15.84:0/4074538751 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3752201497 0 0) 0x7f44680afd10 con 0x7f44680af900 2024-03-29T10:45:48.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.052+0000 7f44667fc640 10 monclient(hunting): _negotiate 2024-03-29T10:45:48.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.052+0000 7f44667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:48.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.052+0000 7f44667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:48.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.052+0000 7f44667fc640 10 monclient(hunting): my global_id is 4602 2024-03-29T10:45:48.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.052+0000 7f44667fc640 1 -- 172.21.15.84:0/4074538751 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44500048a0 con 0x7f44680af900 2024-03-29T10:45:48.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.053+0000 7f44667fc640 1 -- 172.21.15.84:0/4074538751 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 963528675 0 0) 0x7f44500048a0 con 0x7f44680af900 2024-03-29T10:45:48.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.053+0000 7f44667fc640 1 -- 172.21.15.84:0/4074538751 >> v1:172.21.15.84:6790/0 conn(0x7f44680a9ac0 legacy=0x7f44680adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:48.038 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.053+0000 7f44667fc640 1 -- 172.21.15.84:0/4074538751 >> v1:172.21.15.86:6789/0 conn(0x7f446813ee90 legacy=0x7f446813b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:45:48.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.055+0000 7f44667fc640 10 monclient: _finish_hunting 0 2024-03-29T10:45:48.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.055+0000 7f44667fc640 1 monclient: found mon.noname-a 2024-03-29T10:45:48.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.055+0000 7f44667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:48.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.055+0000 7f44667fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.84:6789/0 2024-03-29T10:45:48.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.055+0000 7f44667fc640 1 -- 172.21.15.84:0/4074538751 --> v1:172.21.15.84:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44680aaf60 con 0x7f44680af900 2024-03-29T10:45:48.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.055+0000 7f44667fc640 10 monclient: _finish_auth 0 2024-03-29T10:45:48.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.055+0000 7f44667fc640 10 monclient: _check_auth_tickets 2024-03-29T10:45:48.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.055+0000 7f44667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:48.040 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.055+0000 7f44667fc640 1 -- 172.21.15.84:0/4074538751 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4460001970 con 0x7f44680af900 2024-03-29T10:45:48.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.055+0000 7f44667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:48.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.055+0000 7f44667fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:45:48.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.055+0000 7f44667fc640 10 monclient: dump: 2024-03-29T10:45:48.041 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:48.041 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:48.041 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:48.041 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:48.041 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:48.041 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:48.041 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:48.041 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:48.041 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:48.041 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:48.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.055+0000 7f446dd3d640 5 monclient: authenticate success, global_id 4602 2024-03-29T10:45:48.041 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.055+0000 7f446dd3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:45:48.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.055+0000 7f44667fc640 1 -- 172.21.15.84:0/4074538751 <== mon.0 v1:172.21.15.84:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4460001b50 con 0x7f44680af900 2024-03-29T10:45:48.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.055+0000 7f44667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:48.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.055+0000 7f44667fc640 1 -- 172.21.15.84:0/4074538751 <== mon.0 v1:172.21.15.84:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4460001fa0 con 0x7f44680af900 2024-03-29T10:45:48.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.055+0000 7f44667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:48.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.055+0000 7f44667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:48.042 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.055+0000 7f44667fc640 10 monclient: dump: 2024-03-29T10:45:48.042 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:48.042 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:48.042 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:48.042 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:48.042 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:48.042 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:48.042 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:48.042 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:48.043 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:48.043 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:48.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.055+0000 7f446dd3d640 10 monclient: get_monmap_and_config success 2024-03-29T10:45:48.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.055+0000 7f446dd3d640 4 set_mon_vals no callback set 2024-03-29T10:45:48.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.056+0000 7f446dd3d640 10 monclient: shutdown 2024-03-29T10:45:48.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.056+0000 7f446dd3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:48.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.056+0000 7f446dd3d640 1 -- 172.21.15.84:0/4074538751 >> v1:172.21.15.84:6789/0 conn(0x7f44680af900 legacy=0x7f4468149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:48.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.056+0000 7f446dd3d640 1 -- 172.21.15.84:0/4074538751 shutdown_connections 2024-03-29T10:45:48.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.056+0000 7f446dd3d640 1 -- 172.21.15.84:0/4074538751 >> 172.21.15.84:0/4074538751 conn(0x7f44680b35e0 msgr2=0x7f44680b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:48.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.057+0000 7f446dd3d640 1 -- 172.21.15.84:0/4074538751 shutdown_connections 2024-03-29T10:45:48.043 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.057+0000 7f446dd3d640 1 -- 172.21.15.84:0/4074538751 wait complete. 2024-03-29T10:45:48.054 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.058+0000 7f446dd3d640 10 monclient: build_initial_monmap 2024-03-29T10:45:48.054 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.058+0000 7f446dd3d640 10 monclient: monmap: 2024-03-29T10:45:48.054 INFO:teuthology.misc.health.smithi084.stderr:epoch 0 2024-03-29T10:45:48.054 INFO:teuthology.misc.health.smithi084.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:45:48.054 INFO:teuthology.misc.health.smithi084.stderr:last_changed 0.000000 2024-03-29T10:45:48.054 INFO:teuthology.misc.health.smithi084.stderr:created 0.000000 2024-03-29T10:45:48.054 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 0 (unknown) 2024-03-29T10:45:48.054 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:48.054 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.noname-a 2024-03-29T10:45:48.054 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.noname-b 2024-03-29T10:45:48.054 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.noname-c 2024-03-29T10:45:48.054 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:48.055 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.058+0000 7f446dd3d640 1 librados: starting msgr at 2024-03-29T10:45:48.055 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.058+0000 7f446dd3d640 1 librados: starting objecter 2024-03-29T10:45:48.055 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.058+0000 7f446dd3d640 1 Processor -- start 2024-03-29T10:45:48.055 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.059+0000 7f446dd3d640 1 -- start start 2024-03-29T10:45:48.055 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.059+0000 7f446dd3d640 1 librados: setting wanted keys 2024-03-29T10:45:48.055 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.059+0000 7f446dd3d640 1 librados: calling monclient init 2024-03-29T10:45:48.055 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.059+0000 7f446dd3d640 10 monclient: init 2024-03-29T10:45:48.055 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.059+0000 7f446dd3d640 10 monclient: _reopen_session rank -1 2024-03-29T10:45:48.055 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.059+0000 7f446dd3d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:45:48.055 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.059+0000 7f446dd3d640 10 monclient(hunting): picked mon.noname-c con 0x7f44680a9ac0 addr v1:172.21.15.84:6790/0 2024-03-29T10:45:48.055 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.059+0000 7f446dd3d640 10 monclient(hunting): picked mon.noname-b con 0x7f44680af900 addr v1:172.21.15.86:6789/0 2024-03-29T10:45:48.055 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.061+0000 7f446dd3d640 10 monclient(hunting): picked mon.noname-a con 0x7f446813ee90 addr v1:172.21.15.84:6789/0 2024-03-29T10:45:48.055 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.061+0000 7f446dd3d640 1 -- --> v1:172.21.15.84:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4468141dc0 con 0x7f446813ee90 2024-03-29T10:45:48.055 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.061+0000 7f446dd3d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4468142250 con 0x7f44680af900 2024-03-29T10:45:48.056 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.061+0000 7f446dd3d640 1 -- --> v1:172.21.15.84:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44681ca230 con 0x7f44680a9ac0 2024-03-29T10:45:48.056 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.061+0000 7f446dd3d640 10 monclient(hunting): _renew_subs 2024-03-29T10:45:48.056 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.061+0000 7f446dd3d640 10 monclient(hunting): authenticate will time out at 1450.324447s 2024-03-29T10:45:48.056 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.061+0000 7f4467fff640 1 --1- >> v1:172.21.15.84:6789/0 conn(0x7f446813ee90 0x7f4468141290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.84:6789/0 says I am v1:172.21.15.84:58292/0 (socket says 172.21.15.84:58292) 2024-03-29T10:45:48.056 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.061+0000 7f4467fff640 1 -- 172.21.15.84:0/4226315472 learned_addr learned my addr 172.21.15.84:0/4226315472 (peer_addr_for_me v1:172.21.15.84:0/0) 2024-03-29T10:45:48.056 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.061+0000 7f444ffff640 10 client.?.objecter ms_handle_connect 0x7f446813ee90 2024-03-29T10:45:48.056 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.061+0000 7f444ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:48.056 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.062+0000 7f444ffff640 1 -- 172.21.15.84:0/4226315472 <== mon.0 v1:172.21.15.84:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 907032668 0 0) 0x7f4468141dc0 con 0x7f446813ee90 2024-03-29T10:45:48.056 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.062+0000 7f444ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:48.056 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.062+0000 7f444ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:48.056 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.062+0000 7f444ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:48.056 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.062+0000 7f444ffff640 10 monclient(hunting): my global_id is 4605 2024-03-29T10:45:48.056 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.062+0000 7f444ffff640 1 -- 172.21.15.84:0/4226315472 --> v1:172.21.15.84:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4454004eb0 con 0x7f446813ee90 2024-03-29T10:45:48.056 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.062+0000 7f444ffff640 10 client.?.objecter ms_handle_connect 0x7f44680a9ac0 2024-03-29T10:45:48.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.062+0000 7f444ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:45:48.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.062+0000 7f444ffff640 1 -- 172.21.15.84:0/4226315472 <== mon.2 v1:172.21.15.84:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3334393745 0 0) 0x7f44681ca230 con 0x7f44680a9ac0 2024-03-29T10:45:48.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.062+0000 7f444ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:45:48.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.062+0000 7f444ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:45:48.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.062+0000 7f444ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:45:48.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.062+0000 7f444ffff640 10 monclient(hunting): my global_id is 4709 2024-03-29T10:45:48.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.062+0000 7f444ffff640 1 -- 172.21.15.84:0/4226315472 --> v1:172.21.15.84:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4468141dc0 con 0x7f44680a9ac0 2024-03-29T10:45:48.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.062+0000 7f444ffff640 1 -- 172.21.15.84:0/4226315472 <== mon.2 v1:172.21.15.84:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 367195058 0 0) 0x7f4468141dc0 con 0x7f44680a9ac0 2024-03-29T10:45:48.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.062+0000 7f444ffff640 1 -- 172.21.15.84:0/4226315472 --> v1:172.21.15.84:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44681ca230 con 0x7f44680a9ac0 2024-03-29T10:45:48.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.062+0000 7f444ffff640 1 -- 172.21.15.84:0/4226315472 <== mon.2 v1:172.21.15.84:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f445c0010f0 con 0x7f44680a9ac0 2024-03-29T10:45:48.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.062+0000 7f444ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:48.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.062+0000 7f444ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:45:48.057 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.062+0000 7f444ffff640 10 monclient(hunting): dump: 2024-03-29T10:45:48.057 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:48.057 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:48.058 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:48.058 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:48.058 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:48.058 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:48.058 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:48.058 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:48.058 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:48.058 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:48.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.062+0000 7f444ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:45:48.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.062+0000 7f444ffff640 1 -- 172.21.15.84:0/4226315472 <== mon.0 v1:172.21.15.84:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3413950406 0 0) 0x7f4454004eb0 con 0x7f446813ee90 2024-03-29T10:45:48.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.062+0000 7f444ffff640 1 -- 172.21.15.84:0/4226315472 --> v1:172.21.15.84:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4468141dc0 con 0x7f446813ee90 2024-03-29T10:45:48.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.062+0000 7f444ffff640 1 -- 172.21.15.84:0/4226315472 <== mon.0 v1:172.21.15.84:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f4460001fa0 con 0x7f446813ee90 2024-03-29T10:45:48.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.062+0000 7f444ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:48.058 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.062+0000 7f444ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:45:48.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.062+0000 7f444ffff640 10 monclient(hunting): dump: 2024-03-29T10:45:48.059 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:48.059 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:48.059 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:48.059 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:48.059 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:48.059 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:48.059 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:48.059 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:48.059 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:48.059 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:48.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.062+0000 7f444ffff640 1 -- 172.21.15.84:0/4226315472 <== mon.2 v1:172.21.15.84:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3085554438 0 0) 0x7f44681ca230 con 0x7f44680a9ac0 2024-03-29T10:45:48.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.062+0000 7f444ffff640 1 -- 172.21.15.84:0/4226315472 >> v1:172.21.15.86:6789/0 conn(0x7f44680af900 legacy=0x7f4468140b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:45:48.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.063+0000 7f444ffff640 1 -- 172.21.15.84:0/4226315472 >> v1:172.21.15.84:6789/0 conn(0x7f446813ee90 legacy=0x7f4468141290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:48.059 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.063+0000 7f444ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:45:48.060 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.063+0000 7f444ffff640 1 monclient: found mon.c 2024-03-29T10:45:48.060 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.063+0000 7f444ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:45:48.060 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.063+0000 7f444ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:45:48.060 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.063+0000 7f444ffff640 1 -- 172.21.15.84:0/4226315472 --> v1:172.21.15.84:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44681ca510 con 0x7f44680a9ac0 2024-03-29T10:45:48.060 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.063+0000 7f444ffff640 10 monclient: _finish_auth 0 2024-03-29T10:45:48.060 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.063+0000 7f444ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:45:48.060 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.063+0000 7f444ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:45:48.060 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.064+0000 7f446dd3d640 5 monclient: authenticate success, global_id 4709 2024-03-29T10:45:48.060 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.064+0000 7f446dd3d640 10 monclient: _renew_subs 2024-03-29T10:45:48.060 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.064+0000 7f446dd3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:45:48.060 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.064+0000 7f446dd3d640 1 -- 172.21.15.84:0/4226315472 --> v1:172.21.15.84:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44681ca7f0 con 0x7f44680a9ac0 2024-03-29T10:45:48.060 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.064+0000 7f446dd3d640 10 client.4709.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:45:48.060 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.064+0000 7f446dd3d640 10 monclient: _renew_subs 2024-03-29T10:45:48.060 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.064+0000 7f446dd3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:45:48.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.064+0000 7f446dd3d640 1 -- 172.21.15.84:0/4226315472 --> v1:172.21.15.84:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f44681c8120 con 0x7f44680a9ac0 2024-03-29T10:45:48.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.064+0000 7f446dd3d640 1 librados: init done 2024-03-29T10:45:48.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.065+0000 7f446dd3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:48.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.065+0000 7f446dd3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:48.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.065+0000 7f446dd3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:45:48.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.066+0000 7f446dd3d640 1 -- 172.21.15.84:0/4226315472 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f446813bcd0 con 0x7f44680a9ac0 2024-03-29T10:45:48.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.066+0000 7f444ffff640 1 -- 172.21.15.84:0/4226315472 <== mon.2 v1:172.21.15.84:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f445c0015c0 con 0x7f44680a9ac0 2024-03-29T10:45:48.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.066+0000 7f444ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:45:48.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.066+0000 7f4465ffb640 4 set_mon_vals no callback set 2024-03-29T10:45:48.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.066+0000 7f444ffff640 1 -- 172.21.15.84:0/4226315472 <== mon.2 v1:172.21.15.84:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 78042524 0 0) 0x7f445c001990 con 0x7f44680a9ac0 2024-03-29T10:45:48.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.066+0000 7f444ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:45:48.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.066+0000 7f444ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:45:48.061 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.066+0000 7f444ffff640 10 monclient: dump: 2024-03-29T10:45:48.061 INFO:teuthology.misc.health.smithi084.stderr:epoch 1 2024-03-29T10:45:48.062 INFO:teuthology.misc.health.smithi084.stderr:fsid 88c2d849-612c-4e38-a628-0989e024a981 2024-03-29T10:45:48.062 INFO:teuthology.misc.health.smithi084.stderr:last_changed 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:48.062 INFO:teuthology.misc.health.smithi084.stderr:created 2024-03-29T10:34:13.256565+0000 2024-03-29T10:45:48.062 INFO:teuthology.misc.health.smithi084.stderr:min_mon_release 19 (squid) 2024-03-29T10:45:48.062 INFO:teuthology.misc.health.smithi084.stderr:election_strategy: 1 2024-03-29T10:45:48.062 INFO:teuthology.misc.health.smithi084.stderr:0: v1:172.21.15.84:6789/0 mon.a 2024-03-29T10:45:48.062 INFO:teuthology.misc.health.smithi084.stderr:1: v1:172.21.15.86:6789/0 mon.b 2024-03-29T10:45:48.062 INFO:teuthology.misc.health.smithi084.stderr:2: v1:172.21.15.84:6790/0 mon.c 2024-03-29T10:45:48.062 INFO:teuthology.misc.health.smithi084.stderr: 2024-03-29T10:45:48.062 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.067+0000 7f444ffff640 1 -- 172.21.15.84:0/4226315472 <== mon.2 v1:172.21.15.84:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3323086868 0 0) 0x7f445c001b70 con 0x7f44680a9ac0 2024-03-29T10:45:48.062 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.068+0000 7f444ffff640 1 -- 172.21.15.84:0/4226315472 <== mon.2 v1:172.21.15.84:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1523682299 0 0) 0x7f445c084a50 con 0x7f44680a9ac0 2024-03-29T10:45:48.062 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.068+0000 7f444ffff640 10 client.4709.objecter ms_dispatch 0x7f4468000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T10:45:48.062 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.068+0000 7f444ffff640 3 client.4709.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T10:45:48.062 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.068+0000 7f444ffff640 3 client.4709.objecter handle_osd_map decoding full epoch 11 2024-03-29T10:45:48.063 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.068+0000 7f444ffff640 20 client.4709.objecter dump_active .. 0 homeless 2024-03-29T10:45:48.063 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.068+0000 7f4466ffd640 10 monclient: get_auth_request con 0x7f445406ddf0 auth_method 0 2024-03-29T10:45:48.063 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.068+0000 7f444ffff640 10 client.4709.objecter ms_handle_connect 0x7f445406ddf0 2024-03-29T10:45:48.063 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.077+0000 7f444ffff640 1 -- 172.21.15.84:0/4226315472 <== mon.2 v1:172.21.15.84:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f445c050330 con 0x7f44680a9ac0 2024-03-29T10:45:48.063 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.077+0000 7f444ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:45:48.063 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.077+0000 7f444ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:45:48.370 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.388+0000 7f446dd3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:48.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.388+0000 7f446dd3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:48.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.388+0000 7f446dd3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.84:6790/0 2024-03-29T10:45:48.371 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.388+0000 7f446dd3d640 1 -- 172.21.15.84:0/4226315472 --> v1:172.21.15.84:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4468075f10 con 0x7f44680a9ac0 2024-03-29T10:45:48.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.390+0000 7f444ffff640 1 -- 172.21.15.84:0/4226315472 <== mon.2 v1:172.21.15.84:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 879675878) 0x7f445c052260 con 0x7f44680a9ac0 2024-03-29T10:45:48.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.390+0000 7f444ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:45:48.373 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.390+0000 7f444ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:45:48.374 INFO:teuthology.misc.health.smithi084.stdout: 2024-03-29T10:45:48.374 INFO:teuthology.misc.health.smithi084.stdout:{"epoch":11,"fsid":"88c2d849-612c-4e38-a628-0989e024a981","created":"2024-03-29T10:34:51.337543+0000","modified":"2024-03-29T10:44:56.255219+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:44:56.255219+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8652d97b-41e0-4025-a2e9-9c78b097ffb4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"81f7bc3e-3d25-4774-b43b-8eddffd49bfd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"47ef622f-d2a9-47d5-b13e-3618600661eb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a16650aa-a2da-44e2-bad9-b872009b9413","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0c8cbbef-4bb9-475f-b150-0e97365d0d56","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"93c3ca84-cf2a-4741-b3d1-ca994791a7c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a6752d73-9b2c-451f-92fe-1baa4b29cc9a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"4d1510b3-682c-4fe9-a014-152f64ddbd37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:45:48.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.395+0000 7f446dd3d640 10 librados: watch_flush enter 2024-03-29T10:45:48.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.395+0000 7f446dd3d640 10 librados: watch_flush exit 2024-03-29T10:45:48.379 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.395+0000 7f446dd3d640 20 client.4709.objecter shutdown clearing up homeless session... 2024-03-29T10:45:48.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.395+0000 7f446dd3d640 10 client.4709.objecter successfully canceled tick 2024-03-29T10:45:48.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.395+0000 7f446dd3d640 1 -- 172.21.15.84:0/4226315472 >> v1:172.21.15.86:6816/3337225337 conn(0x7f445406ddf0 legacy=0x7f44540702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:48.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.395+0000 7f446dd3d640 10 monclient: shutdown 2024-03-29T10:45:48.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.395+0000 7f446dd3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:45:48.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.395+0000 7f446dd3d640 1 -- 172.21.15.84:0/4226315472 >> v1:172.21.15.84:6790/0 conn(0x7f44680a9ac0 legacy=0x7f4468071d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:45:48.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.395+0000 7f446dd3d640 1 -- 172.21.15.84:0/4226315472 shutdown_connections 2024-03-29T10:45:48.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.395+0000 7f446dd3d640 1 -- 172.21.15.84:0/4226315472 >> 172.21.15.84:0/4226315472 conn(0x7f44680b35e0 msgr2=0x7f446813dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:45:48.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.396+0000 7f446dd3d640 1 -- 172.21.15.84:0/4226315472 shutdown_connections 2024-03-29T10:45:48.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.396+0000 7f446dd3d640 1 -- 172.21.15.84:0/4226315472 wait complete. 2024-03-29T10:45:48.380 INFO:teuthology.misc.health.smithi084.stderr:2024-03-29T10:45:48.396+0000 7f446dd3d640 1 librados: shutdown 2024-03-29T10:45:48.397 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:45:48.397 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T10:45:48.399 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T10:45:48.399 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T10:45:48.399 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:45:48.437 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T10:45:48.437 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T10:45:48.437 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:45:48.513 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T10:45:48.513 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T10:45:48.513 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:45:48.589 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T10:45:48.590 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T10:45:48.590 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:45:48.659 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T10:45:48.659 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T10:45:48.660 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:45:48.696 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T10:45:48.696 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T10:45:48.697 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:45:48.765 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T10:45:48.766 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T10:45:48.766 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:45:48.838 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T10:45:48.839 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T10:45:48.839 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:45:48.907 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T10:45:48.908 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T10:45:48.908 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T10:45:48.908 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:45:48.932 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T10:45:48.932 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T10:45:48.932 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:45:48.972 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T10:45:48.972 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T10:45:48.973 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T10:45:48.973 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:45:48.984 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T10:45:48.984 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T10:45:48.985 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:45:49.036 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T10:45:49.036 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T10:45:49.036 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T10:45:49.047 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T10:45:49.047 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T10:45:49.047 DEBUG:teuthology.orchestra.run.smithi084:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T10:45:49.149 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-29T10:45:49.149 DEBUG:teuthology.orchestra.run.smithi084:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T10:45:50.191 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-29T10:45:50.191 DEBUG:teuthology.orchestra.run.smithi084:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T10:45:50.294 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-29T10:45:50.294 DEBUG:teuthology.orchestra.run.smithi084:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T10:45:50.352 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi084.front.sepia.ceph.com 2024-03-29T10:45:50.353 DEBUG:teuthology.orchestra.run.smithi084:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T10:45:50.410 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi086.front.sepia.ceph.com 2024-03-29T10:45:50.410 DEBUG:teuthology.orchestra.run.smithi086:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T10:45:53.963 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi086.front.sepia.ceph.com 2024-03-29T10:45:53.964 DEBUG:teuthology.orchestra.run.smithi086:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T10:45:54.059 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi086.front.sepia.ceph.com 2024-03-29T10:45:54.059 DEBUG:teuthology.orchestra.run.smithi086:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T10:45:54.131 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi086.front.sepia.ceph.com 2024-03-29T10:45:54.132 DEBUG:teuthology.orchestra.run.smithi086:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T10:45:54.198 INFO:tasks.ceph:Archiving mon data... 2024-03-29T10:45:54.200 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622642/data/mon.a.tgz 2024-03-29T10:45:54.200 DEBUG:teuthology.orchestra.run.smithi084:> mktemp 2024-03-29T10:45:54.218 INFO:teuthology.orchestra.run.smithi084.stdout:/tmp/tmp.UcDislJ2Vw 2024-03-29T10:45:54.219 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.UcDislJ2Vw 2024-03-29T10:45:54.327 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0666 /tmp/tmp.UcDislJ2Vw 2024-03-29T10:45:54.416 DEBUG:teuthology.orchestra.remote:smithi084:/tmp/tmp.UcDislJ2Vw is 126KB 2024-03-29T10:45:54.469 DEBUG:teuthology.orchestra.run.smithi084:> rm -fr /tmp/tmp.UcDislJ2Vw 2024-03-29T10:45:54.484 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622642/data/mon.c.tgz 2024-03-29T10:45:54.484 DEBUG:teuthology.orchestra.run.smithi084:> mktemp 2024-03-29T10:45:54.539 INFO:teuthology.orchestra.run.smithi084.stdout:/tmp/tmp.iIGbTkbG5L 2024-03-29T10:45:54.540 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.iIGbTkbG5L 2024-03-29T10:45:54.628 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0666 /tmp/tmp.iIGbTkbG5L 2024-03-29T10:45:54.714 DEBUG:teuthology.orchestra.remote:smithi084:/tmp/tmp.iIGbTkbG5L is 126KB 2024-03-29T10:45:54.765 DEBUG:teuthology.orchestra.run.smithi084:> rm -fr /tmp/tmp.iIGbTkbG5L 2024-03-29T10:45:54.780 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622642/data/mon.b.tgz 2024-03-29T10:45:54.780 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2024-03-29T10:45:54.795 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.CIGCawdfhe 2024-03-29T10:45:54.795 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.CIGCawdfhe 2024-03-29T10:45:54.900 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.CIGCawdfhe 2024-03-29T10:45:54.985 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.CIGCawdfhe is 126KB 2024-03-29T10:45:55.008 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.CIGCawdfhe 2024-03-29T10:45:55.023 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T10:45:55.024 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T10:45:55.027 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T10:45:55.107 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T10:45:55.145 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T10:45:55.145 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T10:45:55.147 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622642/remote/smithi084/crash 2024-03-29T10:45:55.148 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T10:45:55.176 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622642/remote/smithi086/crash 2024-03-29T10:45:55.177 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T10:45:55.212 INFO:tasks.ceph:Compressing logs... 2024-03-29T10:45:55.213 DEBUG:teuthology.orchestra.run.smithi084:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T10:45:55.220 DEBUG:teuthology.orchestra.run.smithi086:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T10:45:55.246 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39767.log 2024-03-29T10:45:55.246 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40434.log 2024-03-29T10:45:55.246 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T10:45:55.246 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39767.log: /var/log/ceph/ceph-client.admin.40434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39650.log 2024-03-29T10:45:55.246 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40762.log 2024-03-29T10:45:55.292 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34624.log 2024-03-29T10:45:55.292 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40801.log 2024-03-29T10:45:55.292 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.39650.log: 86.4% 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40434.log.gz 2024-03-29T10:45:55.292 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39767.log.gz 2024-03-29T10:45:55.292 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40762.log: /var/log/ceph/ceph-client.admin.34624.log: /var/log/ceph/ceph-client.admin.40801.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40762.log.gz 2024-03-29T10:45:55.293 INFO:teuthology.orchestra.run.smithi084.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34624.log.gz 2024-03-29T10:45:55.293 INFO:teuthology.orchestra.run.smithi084.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39650.log.gz 2024-03-29T10:45:55.293 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41464.log 2024-03-29T10:45:55.293 INFO:teuthology.orchestra.run.smithi084.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40801.log.gz 2024-03-29T10:45:55.293 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39533.log 2024-03-29T10:45:55.293 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40042.log 2024-03-29T10:45:55.293 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39845.log 2024-03-29T10:45:55.293 INFO:teuthology.orchestra.run.smithi084.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41464.log.gz 2024-03-29T10:45:55.293 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41308.log 2024-03-29T10:45:55.293 INFO:teuthology.orchestra.run.smithi084.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39533.log.gz 2024-03-29T10:45:55.293 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39845.log: gzip -5 85.8% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.40042.log.gz /var/log/ceph/ceph-client.admin.39143.log 2024-03-29T10:45:55.294 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T10:45:55.294 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41308.log: gzip -5 --verbose 85.8% -- -- replaced with /var/log/ceph/ceph-client.admin.39845.log.gz /var/log/ceph/ceph-client.admin.38753.log 2024-03-29T10:45:55.294 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T10:45:55.294 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.39143.log: /var/log/ceph/ceph-client.admin.40996.log 2024-03-29T10:45:55.294 INFO:teuthology.orchestra.run.smithi084.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41308.log.gz 2024-03-29T10:45:55.294 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39806.log 2024-03-29T10:45:55.294 INFO:teuthology.orchestra.run.smithi084.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39143.log.gz 2024-03-29T10:45:55.294 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39221.log 2024-03-29T10:45:55.294 INFO:teuthology.orchestra.run.smithi084.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38753.log.gz 2024-03-29T10:45:55.294 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39806.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40996.log.gz 2024-03-29T10:45:55.294 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38322.log 2024-03-29T10:45:55.294 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39221.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39806.log.gz 2024-03-29T10:45:55.294 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41230.log 2024-03-29T10:45:55.295 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40395.log 2024-03-29T10:45:55.295 INFO:teuthology.orchestra.run.smithi084.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39221.log.gz 2024-03-29T10:45:55.295 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38870.log 2024-03-29T10:45:55.295 INFO:teuthology.orchestra.run.smithi084.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38322.log.gz 2024-03-29T10:45:55.295 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40567.log 2024-03-29T10:45:55.295 INFO:teuthology.orchestra.run.smithi084.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41230.log.gz 2024-03-29T10:45:55.295 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40199.log 2024-03-29T10:45:55.295 INFO:teuthology.orchestra.run.smithi084.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40395.log.gz 2024-03-29T10:45:55.295 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39728.log 2024-03-29T10:45:55.295 INFO:teuthology.orchestra.run.smithi084.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38870.log.gz 2024-03-29T10:45:55.295 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40199.log: 85.1% -- replaced with /var/log/ceph/ceph-client.admin.40567.log.gz 2024-03-29T10:45:55.295 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39416.log 2024-03-29T10:45:55.295 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40918.log 2024-03-29T10:45:55.295 INFO:teuthology.orchestra.run.smithi084.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40199.log.gz 2024-03-29T10:45:55.296 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39416.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39728.log.gz 2024-03-29T10:45:55.296 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T10:45:55.296 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38597.log 2024-03-29T10:45:55.296 INFO:teuthology.orchestra.run.smithi084.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39416.log.gz 2024-03-29T10:45:55.296 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40316.log 2024-03-29T10:45:55.296 INFO:teuthology.orchestra.run.smithi084.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40918.log.gz 2024-03-29T10:45:55.296 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39455.log 2024-03-29T10:45:55.296 INFO:teuthology.orchestra.run.smithi084.stderr: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T10:45:55.296 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40316.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38597.log.gz 2024-03-29T10:45:55.296 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41347.log 2024-03-29T10:45:55.297 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39455.log: gzip -5 --verbose -- 85.9% /var/log/ceph/ceph-client.admin.40684.log -- replaced with /var/log/ceph/ceph-client.admin.40316.log.gz 2024-03-29T10:45:55.297 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T10:45:55.297 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39494.log 2024-03-29T10:45:55.297 INFO:teuthology.orchestra.run.smithi084.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39455.log.gz 2024-03-29T10:45:55.297 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39572.log 2024-03-29T10:45:55.297 INFO:teuthology.orchestra.run.smithi084.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41347.log.gz 2024-03-29T10:45:55.297 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41269.log 2024-03-29T10:45:55.297 INFO:teuthology.orchestra.run.smithi084.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40684.log.gz 2024-03-29T10:45:55.297 INFO:teuthology.orchestra.run.smithi084.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39494.log.gz 2024-03-29T10:45:55.297 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40159.log 2024-03-29T10:45:55.297 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38244.log 2024-03-29T10:45:55.297 INFO:teuthology.orchestra.run.smithi084.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39572.log.gz 2024-03-29T10:45:55.297 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39884.log 2024-03-29T10:45:55.297 INFO:teuthology.orchestra.run.smithi084.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41269.log.gz 2024-03-29T10:45:55.298 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39182.log 2024-03-29T10:45:55.298 INFO:teuthology.orchestra.run.smithi084.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40159.log.gz 2024-03-29T10:45:55.298 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41542.log 2024-03-29T10:45:55.298 INFO:teuthology.orchestra.run.smithi084.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38244.log.gz 2024-03-29T10:45:55.298 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39260.log 2024-03-29T10:45:55.298 INFO:teuthology.orchestra.run.smithi084.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39884.log.gz 2024-03-29T10:45:55.298 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40840.log 2024-03-29T10:45:55.298 INFO:teuthology.orchestra.run.smithi084.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39182.log.gz 2024-03-29T10:45:55.298 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39689.log 2024-03-29T10:45:55.298 INFO:teuthology.orchestra.run.smithi084.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41542.log.gz 2024-03-29T10:45:55.298 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40840.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T10:45:55.298 INFO:teuthology.orchestra.run.smithi084.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39260.log.gz 2024-03-29T10:45:55.298 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41659.log 2024-03-29T10:45:55.298 INFO:teuthology.orchestra.run.smithi084.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40840.log.gz 2024-03-29T10:45:55.299 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39964.log 2024-03-29T10:45:55.299 INFO:teuthology.orchestra.run.smithi084.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39689.log.gz 2024-03-29T10:45:55.299 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41659.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28805.log 2024-03-29T10:45:55.299 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40003.log 2024-03-29T10:45:55.299 INFO:teuthology.orchestra.run.smithi084.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41659.log.gz 2024-03-29T10:45:55.299 INFO:teuthology.orchestra.run.smithi084.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39964.log.gz 2024-03-29T10:45:55.299 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph.tmp-client.admin.28805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38948.log 2024-03-29T10:45:55.299 INFO:teuthology.orchestra.run.smithi084.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28805.log.gz 2024-03-29T10:45:55.299 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39338.log 2024-03-29T10:45:55.299 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38948.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40003.log.gz 2024-03-29T10:45:55.299 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40120.log 2024-03-29T10:45:55.299 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39377.log 2024-03-29T10:45:55.299 INFO:teuthology.orchestra.run.smithi084.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38948.log.gz/var/log/ceph/ceph-client.admin.39338.log: 2024-03-29T10:45:55.299 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40238.log 2024-03-29T10:45:55.300 INFO:teuthology.orchestra.run.smithi084.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39338.log.gz 2024-03-29T10:45:55.300 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39377.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40120.log.gz 2024-03-29T10:45:55.300 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40488.log 2024-03-29T10:45:55.300 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40355.log 2024-03-29T10:45:55.300 INFO:teuthology.orchestra.run.smithi084.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.39377.log.gz 2024-03-29T10:45:55.300 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40238.log: /var/log/ceph/ceph-client.admin.40488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39299.log 2024-03-29T10:45:55.300 INFO:teuthology.orchestra.run.smithi084.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40238.log.gz 2024-03-29T10:45:55.300 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41698.log 2024-03-29T10:45:55.300 INFO:teuthology.orchestra.run.smithi084.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40488.log.gz 2024-03-29T10:45:55.300 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39299.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40355.log.gz 2024-03-29T10:45:55.300 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T10:45:55.300 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40277.log 2024-03-29T10:45:55.300 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41698.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.39299.log.gz 2024-03-29T10:45:55.301 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38400.log 2024-03-29T10:45:55.301 INFO:teuthology.orchestra.run.smithi084.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41698.log.gz 2024-03-29T10:45:55.301 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38909.log 2024-03-29T10:45:55.301 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38831.log 2024-03-29T10:45:55.301 INFO:teuthology.orchestra.run.smithi084.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40277.log.gz 2024-03-29T10:45:55.301 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41113.log 2024-03-29T10:45:55.301 INFO:teuthology.orchestra.run.smithi084.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38400.log.gz 2024-03-29T10:45:55.301 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41035.log 2024-03-29T10:45:55.301 INFO:teuthology.orchestra.run.smithi084.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38909.log.gz 2024-03-29T10:45:55.301 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41620.log 2024-03-29T10:45:55.301 INFO:teuthology.orchestra.run.smithi084.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38831.log.gz 2024-03-29T10:45:55.301 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41035.log: 86.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38519.log 2024-03-29T10:45:55.301 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41113.log.gz 2024-03-29T10:45:55.301 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38714.log 2024-03-29T10:45:55.302 INFO:teuthology.orchestra.run.smithi084.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41035.log.gz 2024-03-29T10:45:55.302 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39065.log 2024-03-29T10:45:55.302 INFO:teuthology.orchestra.run.smithi084.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41620.log.gz 2024-03-29T10:45:55.302 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38714.log: gzip -5 --verbose -- 86.0% /var/log/ceph/ceph-client.admin.41503.log 2024-03-29T10:45:55.302 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38519.log.gz 2024-03-29T10:45:55.302 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39065.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38714.log.gz 2024-03-29T10:45:55.302 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40606.log 2024-03-29T10:45:55.302 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40645.log 2024-03-29T10:45:55.302 INFO:teuthology.orchestra.run.smithi084.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39065.log.gz 2024-03-29T10:45:55.302 INFO:teuthology.orchestra.run.smithi084.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41503.log.gz 2024-03-29T10:45:55.302 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40879.log 2024-03-29T10:45:55.302 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41191.log 2024-03-29T10:45:55.302 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40645.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40606.log.gz 2024-03-29T10:45:55.303 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40879.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T10:45:55.303 INFO:teuthology.orchestra.run.smithi084.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40645.log.gz 2024-03-29T10:45:55.303 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41191.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40879.log.gz 2024-03-29T10:45:55.303 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41386.log 2024-03-29T10:45:55.303 INFO:teuthology.orchestra.run.smithi084.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41191.log.gz 2024-03-29T10:45:55.303 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38478.log 2024-03-29T10:45:55.303 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41581.log 2024-03-29T10:45:55.303 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41386.log: /var/log/ceph/ceph-client.admin.38478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40527.log 2024-03-29T10:45:55.303 INFO:teuthology.orchestra.run.smithi084.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41386.log.gz 2024-03-29T10:45:55.303 INFO:teuthology.orchestra.run.smithi084.stderr:gzip 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38478.log.gz 2024-03-29T10:45:55.303 INFO:teuthology.orchestra.run.smithi084.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38439.log 2024-03-29T10:45:55.303 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41581.log: /var/log/ceph/ceph-client.admin.40527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41074.log 2024-03-29T10:45:55.303 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38439.log: 86.8% 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40527.log.gz 2024-03-29T10:45:55.303 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41581.log.gz 2024-03-29T10:45:55.303 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34915.log 2024-03-29T10:45:55.304 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41074.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38439.log.gz 2024-03-29T10:45:55.304 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41425.log 2024-03-29T10:45:55.304 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.34915.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41074.log.gz 2024-03-29T10:45:55.304 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39923.log 2024-03-29T10:45:55.304 INFO:teuthology.orchestra.run.smithi084.stderr:gzip/var/log/ceph/ceph-client.admin.41425.log: -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T10:45:55.304 INFO:teuthology.orchestra.run.smithi084.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.34915.log.gz 2024-03-29T10:45:55.304 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40723.log 2024-03-29T10:45:55.304 INFO:teuthology.orchestra.run.smithi084.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41425.log.gz 2024-03-29T10:45:55.304 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39611.log 2024-03-29T10:45:55.304 INFO:teuthology.orchestra.run.smithi084.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39923.log.gz 2024-03-29T10:45:55.304 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.40723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38792.log 2024-03-29T10:45:55.304 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39611.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40723.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39026.log 2024-03-29T10:45:55.304 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T10:45:55.305 INFO:teuthology.orchestra.run.smithi084.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39611.log.gz 2024-03-29T10:45:55.305 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38987.log 2024-03-29T10:45:55.305 INFO:teuthology.orchestra.run.smithi084.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38792.log.gz 2024-03-29T10:45:55.305 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39026.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T10:45:55.305 INFO:teuthology.orchestra.run.smithi084.stderr: 85.9%/var/log/ceph/ceph-client.admin.38987.log: -- replaced with /var/log/ceph/ceph-client.admin.39026.log.gz 2024-03-29T10:45:55.305 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38283.log 2024-03-29T10:45:55.305 INFO:teuthology.orchestra.run.smithi084.stderr: 85.9%gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T10:45:55.305 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38987.log.gz 2024-03-29T10:45:55.305 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.38283.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38283.log.gz 2024-03-29T10:45:55.306 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39104.log 2024-03-29T10:45:55.306 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-osd.2.log: 92.8% -- replaced with /var/log/ceph/ceph.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38558.log 2024-03-29T10:45:55.306 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T10:45:55.306 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41152.log 2024-03-29T10:45:55.306 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.39104.log: /var/log/ceph/ceph-client.admin.38558.log: 87.5% 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39104.log.gz 2024-03-29T10:45:55.306 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38558.log.gz 2024-03-29T10:45:55.306 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40081.log 2024-03-29T10:45:55.307 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38361.log 2024-03-29T10:45:55.307 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.41152.log: /var/log/ceph/ceph-client.admin.40081.log: 93.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T10:45:55.307 INFO:teuthology.orchestra.run.smithi084.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41152.log.gz 2024-03-29T10:45:55.307 INFO:teuthology.orchestra.run.smithi084.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40081.log.gz 2024-03-29T10:45:55.307 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40957.log 2024-03-29T10:45:55.307 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38675.log 2024-03-29T10:45:55.308 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.40957.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38361.log.gz 2024-03-29T10:45:55.308 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35018.log 2024-03-29T10:45:55.308 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.38675.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40957.log.gz 2024-03-29T10:45:55.308 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38636.log 2024-03-29T10:45:55.308 INFO:teuthology.orchestra.run.smithi084.stderr: 86.0%gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T10:45:55.309 INFO:teuthology.orchestra.run.smithi084.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38675.log.gz 2024-03-29T10:45:55.309 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/ceph-client.admin.35018.log: /var/log/ceph/ceph-client.admin.38636.log: /var/log/ceph/ceph-mon.c.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35018.log.gz 2024-03-29T10:45:55.309 INFO:teuthology.orchestra.run.smithi084.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38636.log.gz 2024-03-29T10:45:55.316 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T10:45:55.316 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34405.log 2024-03-29T10:45:55.316 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34365.log 2024-03-29T10:45:55.316 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34405.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34605.log 2024-03-29T10:45:55.316 INFO:teuthology.orchestra.run.smithi086.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34405.log.gz 2024-03-29T10:45:55.316 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34645.log 2024-03-29T10:45:55.317 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34605.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T10:45:55.317 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34645.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34365.log.gz 2024-03-29T10:45:55.317 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T10:45:55.317 INFO:teuthology.orchestra.run.smithi086.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34605.log.gz 2024-03-29T10:45:55.317 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34485.log 2024-03-29T10:45:55.317 INFO:teuthology.orchestra.run.smithi086.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34645.log.gz 2024-03-29T10:45:55.317 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34445.log 2024-03-29T10:45:55.318 INFO:teuthology.orchestra.run.smithi086.stderr: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T10:45:55.318 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5/var/log/ceph/ceph-client.admin.34485.log: --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T10:45:55.318 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34445.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T10:45:55.318 INFO:teuthology.orchestra.run.smithi086.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34485.log.gz 2024-03-29T10:45:55.318 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34565.log 2024-03-29T10:45:55.318 INFO:teuthology.orchestra.run.smithi086.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34445.log.gz 2024-03-29T10:45:55.319 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T10:45:55.319 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34565.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T10:45:55.319 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34565.log.gz 2024-03-29T10:45:55.319 INFO:teuthology.orchestra.run.smithi086.stderr: --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T10:45:55.319 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34525.log 2024-03-29T10:45:55.319 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph.log: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T10:45:55.319 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34525.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34525.log.gz 2024-03-29T10:45:55.361 INFO:teuthology.orchestra.run.smithi086.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T10:46:07.359 INFO:teuthology.orchestra.run.smithi084.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T10:46:07.421 INFO:teuthology.orchestra.run.smithi086.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T10:46:07.441 INFO:teuthology.orchestra.run.smithi086.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T10:46:07.563 INFO:teuthology.orchestra.run.smithi086.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T10:46:11.475 INFO:teuthology.orchestra.run.smithi084.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T10:46:11.650 INFO:teuthology.orchestra.run.smithi086.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T10:46:12.946 INFO:teuthology.orchestra.run.smithi084.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T10:46:12.977 INFO:teuthology.orchestra.run.smithi084.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T10:46:41.964 INFO:teuthology.orchestra.run.smithi084.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T10:46:52.487 INFO:teuthology.orchestra.run.smithi084.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T10:46:52.488 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-03-29T10:46:52.488 INFO:teuthology.orchestra.run.smithi084.stderr:real 0m57.256s 2024-03-29T10:46:52.489 INFO:teuthology.orchestra.run.smithi084.stderr:user 2m41.632s 2024-03-29T10:46:52.489 INFO:teuthology.orchestra.run.smithi084.stderr:sys 0m5.656s 2024-03-29T10:47:04.537 INFO:teuthology.orchestra.run.smithi086.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T10:47:04.539 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T10:47:04.539 INFO:teuthology.orchestra.run.smithi086.stderr:real 1m9.266s 2024-03-29T10:47:04.539 INFO:teuthology.orchestra.run.smithi086.stderr:user 1m57.665s 2024-03-29T10:47:04.539 INFO:teuthology.orchestra.run.smithi086.stderr:sys 0m4.380s 2024-03-29T10:47:04.539 INFO:tasks.ceph:Archiving logs... 2024-03-29T10:47:04.539 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622642/remote/smithi084/log 2024-03-29T10:47:04.541 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T10:47:28.093 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622642/remote/smithi086/log 2024-03-29T10:47:28.094 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T10:47:46.158 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T10:47:46.406 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=299fe7904f064c6e85fc5cc2020f1ea1 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T10:47:46.411 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T10:47:46.420 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T10:47:46.437 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T10:47:46.437 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T10:47:46.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T10:47:46.541 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T10:47:46.576 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T10:47:46.577 DEBUG:teuthology.orchestra.run.smithi084:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:47:46.580 DEBUG:teuthology.orchestra.run.smithi086:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:47:46.594 INFO:teuthology.orchestra.run.smithi084.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:47:46.596 INFO:teuthology.orchestra.run.smithi086.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:47:46.599 INFO:teuthology.orchestra.run.smithi084.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:47:46.599 INFO:teuthology.orchestra.run.smithi084.stdout:=============================================================================== 2024-03-29T10:47:46.600 INFO:teuthology.orchestra.run.smithi084.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 48 +468us[ +468us] +/- 66ms 2024-03-29T10:47:46.600 INFO:teuthology.orchestra.run.smithi084.stdout:^* hv02.front.sepia.ceph.com 3 6 377 49 -148us[ -149us] +/- 66ms 2024-03-29T10:47:46.600 INFO:teuthology.orchestra.run.smithi084.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 48 -194us[ -194us] +/- 77ms 2024-03-29T10:47:46.600 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:47:46.601 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:47:46.602 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-03-29T10:47:46.602 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 48 +431us[ +476us] +/- 66ms 2024-03-29T10:47:46.602 INFO:teuthology.orchestra.run.smithi086.stdout:^* hv02.front.sepia.ceph.com 3 6 377 47 -185us[ -139us] +/- 66ms 2024-03-29T10:47:46.602 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 45 -223us[ -223us] +/- 77ms 2024-03-29T10:47:46.602 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:47:46.602 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T10:47:46.625 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T10:47:46.626 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T10:47:46.646 DEBUG:teuthology.orchestra.run.smithi084:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T10:47:46.718 DEBUG:teuthology.orchestra.run.smithi086:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T10:47:46.789 DEBUG:teuthology.orchestra.run.smithi084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:47:46.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:47:46.822 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:47:46.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:47:46.856 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T10:47:46.949 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T10:47:46.964 INFO:teuthology.task.internal:Duration was 1226.148806 seconds 2024-03-29T10:47:46.965 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T10:47:46.973 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T10:47:46.973 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T10:47:46.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T10:47:47.022 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:47:47.024 INFO:teuthology.orchestra.run.smithi084.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:47:47.485 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T10:47:47.486 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi084.front.sepia.ceph.com 2024-03-29T10:47:47.487 DEBUG:teuthology.orchestra.run.smithi084:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T10:47:47.510 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi086.front.sepia.ceph.com 2024-03-29T10:47:47.510 DEBUG:teuthology.orchestra.run.smithi086:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T10:47:47.532 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T10:47:47.533 DEBUG:teuthology.orchestra.run.smithi084:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T10:47:47.553 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T10:47:47.604 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T10:47:47.604 DEBUG:teuthology.orchestra.run.smithi084:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T10:47:47.625 DEBUG:teuthology.orchestra.run.smithi086:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T10:47:47.742 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T10:47:47.772 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T10:47:47.772 DEBUG:teuthology.orchestra.run.smithi084:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T10:47:47.799 DEBUG:teuthology.orchestra.run.smithi086:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T10:47:47.826 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T10:47:47.839 DEBUG:teuthology.orchestra.run.smithi084:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:47:47.842 DEBUG:teuthology.orchestra.run.smithi086:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:47:47.865 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern = core 2024-03-29T10:47:47.892 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = core 2024-03-29T10:47:47.930 DEBUG:teuthology.orchestra.run.smithi084:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:47:47.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:47:47.947 DEBUG:teuthology.orchestra.run.smithi086:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T10:47:47.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:47:47.984 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T10:47:48.014 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T10:47:48.015 DEBUG:teuthology.misc:Transferring archived files from smithi084:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622642/remote/smithi084 2024-03-29T10:47:48.015 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T10:47:48.051 DEBUG:teuthology.misc:Transferring archived files from smithi086:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622642/remote/smithi086 2024-03-29T10:47:48.051 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T10:47:48.086 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T10:47:48.086 DEBUG:teuthology.orchestra.run.smithi084:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T10:47:48.089 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T10:47:48.136 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T10:47:48.147 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T10:47:48.148 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T10:47:48.160 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T10:47:48.161 DEBUG:teuthology.orchestra.run.smithi084:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T10:47:48.163 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T10:47:48.175 INFO:teuthology.orchestra.run.smithi084.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 10:47 /home/ubuntu/cephtest 2024-03-29T10:47:48.190 INFO:teuthology.orchestra.run.smithi086.stdout: 265590 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 10:47 /home/ubuntu/cephtest 2024-03-29T10:47:48.191 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T10:47:48.203 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T10:47:48.235 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T10:47:48.265 DEBUG:teuthology.nuke:shortname: smithi084 2024-03-29T10:47:48.265 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:47:48.290 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi084.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622642', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:18:50.477775', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMXl6Z9hk+MhmxBMtW9InRHY/lu+i+im1gzZI1WaqF17XERyUg+tOpwSRjG/+63JuJhTlOL2azv4V46SqomePKE='} 2024-03-29T10:47:48.293 DEBUG:teuthology.nuke:shortname: smithi086 2024-03-29T10:47:48.293 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:47:48.314 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622642', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:18:50.476354', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE0cRLpkX/ZRNRgRxmItsxmFa4K2SATDuO+KjNERXWH7L6Wu1GBemAyH1YSSIGIkyoPht418Dx+hVVZEPMw6gOg='} 2024-03-29T10:47:48.342 INFO:teuthology.orchestra.console.smithi084:Power off 2024-03-29T10:47:48.343 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T10:47:48.366 DEBUG:teuthology.orchestra.console.smithi084:power off output: Chassis Power Control: Down/Off 2024-03-29T10:47:48.367 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:47:48.382 INFO:teuthology.orchestra.console.smithi086:Power off 2024-03-29T10:47:48.382 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T10:47:48.394 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on 2024-03-29T10:47:48.408 DEBUG:teuthology.orchestra.console.smithi086:power off output: Chassis Power Control: Down/Off 2024-03-29T10:47:48.408 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:47:48.430 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-03-29T10:47:52.396 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:47:52.431 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:47:52.506 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is on 2024-03-29T10:47:52.541 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-03-29T10:47:56.510 DEBUG:teuthology.orchestra.console.smithi084:pexpect command: ipmitool -H smithi084.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:47:56.542 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T10:47:56.621 DEBUG:teuthology.orchestra.console.smithi084:check power output: Chassis Power is off 2024-03-29T10:47:56.653 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is off 2024-03-29T10:47:56.721 INFO:teuthology.orchestra.console.smithi084:Power off completed 2024-03-29T10:47:56.754 INFO:teuthology.orchestra.console.smithi086:Power off completed 2024-03-29T10:47:56.861 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} duration: 1226.1488063335419 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=299fe7904f064c6e85fc5cc2020f1ea1 status: fail success: false 2024-03-29T10:47:56.862 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:47:56.976 INFO:teuthology.run:FAIL